DEV Community: sam-nash

Deploy Docker images on AWS

sam-nash — Thu, 12 Dec 2024 11:41:45 +0000

Deploying Docker images on Amazon Elastic Container Registry (ECR) and running them as containers on AWS can be accomplished through several AWS services. Here are the most common and effective options:

1. Amazon Elastic Container Service (ECS)
Overview: ECS is a fully managed container orchestration service. It supports Docker containers and allows you to deploy, manage, and scale applications easily.

Modes:

Fargate: A serverless compute engine that runs containers without needing to manage the underlying infrastructure. It’s a good option if you want to focus on your containers without managing EC2 instances.

EC2: Run containers on EC2 instances you manage. This option gives more control over the underlying infrastructure, allowing you to configure networking, storage, and more.

Integration with ECR: ECS has direct integration with ECR, allowing you to pull images securely and easily. Simply specify the ECR image URI in the ECS task definition.

Use Cases: Suitable for microservices architectures, batch processing, web applications, and serverless deployments where you want AWS to handle much of the infrastructure management.

2. Amazon Elastic Kubernetes Service (EKS)
Overview: EKS is a managed Kubernetes service for running Kubernetes applications on AWS.

Flexibility: EKS offers more flexibility, especially if you are familiar with Kubernetes or have workloads that need Kubernetes features. EKS allows you to leverage Kubernetes' powerful ecosystem for advanced orchestration and customizations.

Integration with ECR: EKS can pull images directly from ECR, leveraging IAM roles for secure access.

Use Cases: Ideal for organizations with experience in Kubernetes, or for applications that need complex orchestration features, hybrid-cloud, or multi-cloud deployments.

3. AWS App Runner
Overview: AWS App Runner is a fully managed service that provides a simple way to deploy containerized web applications or APIs from ECR without managing servers or infrastructure.

Advantages: Minimal setup, fast deployment, and automatic scaling. It handles much of the operational overhead.

Integration with ECR: Connects directly with ECR for deploying images. App Runner automatically manages load balancing and scaling.

Use Cases: Suitable for simpler applications or services (such as web apps and APIs) where you need speed and simplicity in deployment rather than complex orchestration.

4. AWS Lambda (with Docker Image support)
Overview: AWS Lambda allows you to run containerized applications as Lambda functions, which is ideal for event-driven or serverless workloads.

Integration with ECR: Lambda can pull images directly from ECR, allowing you to deploy container images up to 10 GB in size.

Use Cases: Best for microservices that benefit from serverless architecture, short-lived processes, or workloads that don’t require the full orchestration capabilities of ECS or EKS.

Comparison Summary

Service	Best For	Management Level	Scalability
ECS (Fargate)	Microservices, low-management applications	Fully managed	Auto-scalable
ECS (EC2)	More control over infrastructure	Semi-managed	Manual scaling
EKS	Advanced orchestrations, hybrid cloud	Kubernetes-managed	Kubernetes-native
App Runner	Quick web app deployment, low complexity	Fully managed	Auto-scalable
Lambda	Event-driven, serverless functions	Fully managed	Event-based

Terraform Module Sources

sam-nash — Thu, 12 Dec 2024 11:36:51 +0000

How Terraform Finds Reusable Modules

Terraform lets you break down your infrastructure configuration into smaller, reusable pieces called modules. But where does Terraform find the code for these modules? That's where the source argument comes in.

The source argument tells Terraform where to look for the module's code. There are several options available, depending on where you store your modules:

Local Files: Perfect for keeping modules within the same project. Just use a path starting with ./ or ../ to point Terraform to the module directory.

module "local_database" {
  source = "./modules/database"
}

Terraform Registry: This is a terraform platform for discovering providers, modules and security policies. You can specify the module's name and version to use a specific one.

module "aws_ec2_instance" {
  source  = "hashicorp/aws/ec2"
  version = "~> 4.0"
}

Version Control Systems (VCS): If your modules live in a Git repository (like GitHub or Bitbucket), or a Mercurial repository, you can provide the URL to the repository.

module "private_module" {
  source = "git@github.com:your-org/your-private-module.git"
}

Http/s URLs: Modules can also be downloaded from web servers using HTTP or HTTPS URLs. Terraform can even follow redirects to other source locations.

module "http_module" {
  source = "https://your-company-registry.com/modules/my-module"
}

Cloud Storage: Modules stored in Amazon S3 buckets can be accessed by prefixing the URL with s3::

module "s3_module" {
  source = "s3::s3.amazonaws.com/your-bucket/my-module.zip"
}

Terraform Commands Cheat Sheet

sam-nash — Wed, 11 Dec 2024 13:39:55 +0000

This post serves as a comprehensive cheat sheet for commonly used Terraform commands, offering detailed explanations and where possible with some practical examples. Whether you're initializing a workspace, creating execution plans, managing state files, or destroying resources, I think this guide has you covered. Perfect for beginners and seasoned practitioners alike, it ensures you can leverage Terraform to its full potential.

Terraform General Commands

terraform version : Show the Terraform version.
terraform init : Initialize the working directory and download the required providers and plugins.
terraform init -reconfigure : Reconfigure the backend and update the configuration.
terraform init -get-plugins=false : Initialize without downloading plugins (deprecated, as plugins are automatically managed as providers).
terraform init -verify-plugins=false : Initialize without verifying plugin signatures (deprecated, use with caution).
terraform fmt : Format the Terraform configuration files to canonical format.
terraform fmt -recursive : Format all files in the directory and subdirectories recursively.
terraform fmt -check : Check the format of the Terraform configuration files. Does not modify files.
terraform validate : Validate the syntax and consistency of Terraform configuration files.
terraform validate -backend=false : Validate configuration without initializing the backend.
terraform validate -json : Validate configuration and output results in JSON format.

Terraform Plan Commands

terraform plan : Create an execution plan showing the changes Terraform will make.
terraform plan -out=tf_plan.out : Save the generated execution plan to a file.
terraform plan -destroy : Create a plan to destroy all infrastructure.
terraform plan -var-file=my_vars.tfvars : Include variables from the specified file in the plan.
terraform plan -var=my_variable=value : Override variables with specific values for the plan.
terraform plan -parallelism=4 : Limit the number of parallel resource operations (default is 10).

Terraform Apply Commands

terraform apply : Apply changes based on the current configuration.
terraform apply tf_plan.out : Apply changes based on a saved execution plan file.
terraform apply -auto-approve : Apply changes without prompting for approval.
terraform apply -target=resource_address : Apply changes only to the specified resource(s).
terraform apply -var=my_variable=value : Override variables with specific values during apply.
terraform apply -var-file=my_vars.tfvars : Use variables from the specified file during apply.
terraform apply -lock=true : Lock the statefile before applying changes (default behavior).
terraform apply -lock-timeout=20s : Set a timeout for acquiring a state lock.
terraform apply -input=false : Suppress interactive input prompts.
terraform apply -replace=resource_address : Force replacement of a specific resource.
terraform apply -parallelism=4 : Limit the number of concurrent resource operations during apply.
terraform apply -refresh-only : Refresh the state without applying changes (available from Terraform 1.0).

Terraform Destroy Commands

terraform destroy : Destroy all managed infrastructure.
terraform destroy -auto-approve : Destroy resources without prompting for confirmation.
terraform destroy -target=resource_address : Destroy specific resource(s).
terraform destroy -var=my_variable=value : Override variables during destroy.
terraform destroy -var-file=my_vars.tfvars : Use variables from the specified file during destroy.

Terraform State Commands

terraform state list : List all resources in the state file.
terraform state show resource_address : Show the details of a specific resource.
terraform state mv resource_address new_resource_address : Move a resource to a new address.
terraform state rm resource_address : Remove a resource from the state file.
terraform state pull > terraform.tfstate : Download the current state file.
terraform state push terraform.tfstate : Upload a state file to a remote backend.
terraform state replace-provider source_provider target_provider : Replace a provider in the state file.

Terraform Import Commands

terraform import resource_address external_id : Import existing infrastructure into the Terraform state.

Terraform Workspace Commands

terraform workspace list : List all available workspaces.
terraform workspace show : Show the currently selected workspace.
terraform workspace new workspace_name : Create a new workspace.
terraform workspace select workspace_name : Switch to a specific workspace.
Terraform Graph Commands
terraform graph : Generate a visual representation of the dependency graph.
terraform graph -type=plan : Generate a graph based on the execution plan.
terraform graph -type=apply : Generate a graph based on the apply plan.
terraform graph -draw-cycles : Highlight cyclic dependencies in the graph.

Terraform Output Commands

terraform output : Display all output variables.
terraform output -json : Display output variables in JSON format.
terraform output variable_name : Show the value of a specific output variable.

Terraform Show Commands

terraform show : Show the current state or plan.
terraform show -json : Output the state or plan in JSON format.

Other Commands

terraform refresh : Update the state file to reflect the current state of resources.
terraform taint resource_address : Mark a resource for recreation on the next apply.
terraform untaint resource_address : Remove the taint from a resource.
terraform providers : List providers used in the configuration.
terraform providers schema : Display provider schema details (available in recent versions).

Terraform Basics

sam-nash — Wed, 11 Dec 2024 07:49:20 +0000

Terraform is an Infrastructure as Code (IaC) tool that enables you to manage your infrastructure through code instead of manual processes. Using Terraform, you define your infrastructure in configuration files (written primarily in HashiCorp Configuration Language - HCL), specifying the components required to run your applications, such as servers, storage, and networking resources.

Here’s a more detailed breakdown of how Terraform operates:

Configuration Files:

These files define the desired state of your infrastructure, detailing resources like virtual machines, databases, and network settings. Configuration files are written with a declarative style syntax in HashiCorp Configuration Language (HCL) format. These files define the resources you want to create or modify.

Terraform uses the resource type identifiers (e.g., aws_instance for AWS EC2, google_compute_instance for Google Compute Engine) to route the request to the correct provider plugin.

resource "aws_instance" "example" {
    ami           = "ami-0c55b159cbfafe1f0"
    instance_type = "t2.micro"
}

Writing a Terraform Configuration File

1. Structure of Configuration Files

Terraform configuration is split into different sections:

Provider: Specifies the cloud provider or platform (e.g., Google, AWS, Azure) you’re working with.
Resource: Defines the resources to be created or managed.
Variable: Declares inputs that can be reused across the configuration.
Output: Defines outputs to display or pass values after applying changes.

2. Organizing Terraform Files

You can split your Terraform configuration across multiple .tf files in a single directory. Terraform loads all .tf files in the directory and processes them together. Here's a typical structure:

/my-terraform-project
│
├── main.tf           # Main configuration file
├── variables.tf      # Variables declaration
├── outputs.tf        # Outputs declaration
└── terraform.tfvars  # Variables values (optional)

(provider.tf) : Contains the provider information

provider "google" {
  project = "my-project-id"
  region  = "us-west1"
}

(main.tf) : Contains the core configuration.

resource "google_compute_instance" "example" {
  name         = "example-instance"
  machine_type = "e2-medium"
  zone         = "us-west1-a"

  boot_disk {
    initialize_params {
      image = "projects/debian-cloud/global/images/family/debian-11"
    }
  }

  network_interface {
    network = "default"
  }
}

variables.tf: Defines reusable variables.
outputs.tf: Declares outputs for the module.
terraform.tfvars: Stores values for variables (if not provided directly or via environment variables).

3. Declaring Variables

Variables are defined using the variable block in variables.tf:

Example:

variable "project_id" {
  description = "The GCP project ID"
  type        = string
}

variable "machine_type" {
  description = "The type of VM machine"
  type        = string
  default     = "e2-medium"
}

4. Passing Variables

Variables can be passed in several ways:

In the terraform.tfvars file:

project_id = "my-project-id"
machine_type = "e2-standard-4"

Via the command line:

terraform apply -var="project_id=my-project-id"

Using Environment Variables: Set an environment variable with the TF_VAR_ prefix:

export TF_VAR_project_id="my-project-id"

5. Referencing Variables

Variables are accessed using the var keyword:

provider "google" {
  project = var.project_id
  region  = "us-west1"
}

resource "google_compute_instance" "example" {
  name         = "example-instance"
  machine_type = var.machine_type
  zone         = "us-west1-a"

  boot_disk {
    initialize_params {
      image = "projects/debian-cloud/global/images/family/debian-11"
    }
  }

  network_interface {
    network = "default"
  }
}

1. Variable Data Types
String: Represents a single line of text.
Number: Represents a numeric value.
Bool: Represents a boolean value, either true or false.
List: Represents a collection of values in a specific order.
Map: Represents a collection of key-value pairs.
Object: Represents a structured collection of named attributes, each with its own type.
Set: Represents a collection of unique values with no specific order.

Providers:

Terraform uses providers to interact with various cloud services and platforms. Providers are plugins that specify the resources available for a given service. For example, the AWS provider allows Terraform to manage resources like EC2 instances and S3 buckets. Similar providers exist for Google Cloud, Azure, Kubernetes, and more.

Terraform Providers and API Calls

Terraform providers are essentially plugins that allow Terraform to interact with various cloud services, like AWS, Google Cloud, and Azure. These providers are responsible for making the necessary API calls to these services to create, update, or delete resources as specified in your configuration files.

Here’s a step-by-step overview of the process:

Initialization (terraform init): When you run this command, Terraform checks your configuration files for the required providers.
Downloading Providers: Terraform downloads the necessary provider plugins from the Terraform Registry (or other specified sources). These plugins contain the code to interact with the provider’s API.
Authentication: Each provider plugin includes mechanisms for authentication. For example, with AWS, you might need to configure access keys or use IAM roles. The provider plugin handles these details and establishes a secure connection to the API.
API Calls: Once initialized and authenticated, Terraform uses the provider plugins to validate the syntax and translate the configuration into API calls specific to the provider. For example, if your configuration includes an compute instance, the Google provider plugin will make the appropriate API calls to GCP to create the compute instance with the specified parameters.

Example of GCP Provider Initialization

Here’s a snippet of a Terraform configuration that uses the Google provider:

provider "google" {
  project = "your-project-id"
  region  = "us-west1"
}

resource "google_compute_instance" "example" {
  name         = "example-instance"
  machine_type = "e2-micro"
  zone         = "us-west1-a"

  boot_disk {
    initialize_params {
      image = "projects/debian-cloud/global/images/family/debian-11"
    }
  }

  network_interface {
    network       = "default"
    access_config {
      # This is necessary to allow external internet access
    }
  }
}

When you run terraform init the output might look something like below.

Initializing the backend...

Successfully configured the backend "gcs"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Finding latest version of hashicorp/google...
- Installing hashicorp/google v6.11.2...
- Installed hashicorp/google v6.11.2 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

In this example:

When you run terraform init, Terraform will download the Google provider plugin.
This plugin includes the code required to authenticate with Google Cloud and interact with its APIs.
When you run terraform apply, the plugin uses Google Cloud APIs to create and manage resources, such as Compute Engine instances or Cloud Storage buckets, based on your configuration.

Terraform Format

The terraform fmt command automatically formats your Terraform configuration files to follow the standard formatting conventions. This includes aligning indentation, organizing blocks, and correcting syntax spacing. Consistent formatting improves the readability of your code, making it easier for teams to collaborate and review changes. For example, if a configuration file has inconsistent indentation or misplaced brackets, running terraform fmt will fix these issues automatically. Here’s how you use it:

terraform fmt

This command scans all .tf files in the current directory (and subdirectories if specified) and updates them to match the standard format. When formatting is successful, you might see an output that will look like.

main.tf
variables.tf

Terraform validate

The terraform validate command checks the syntax and logical structure of your configuration files for errors. It ensures that the configuration is syntactically correct and that all required arguments are specified. However, it does not interact with the remote provider or check resource availability; it only validates the local files. For example, if you define a google_compute_instance resource but forget to specify a mandatory field like machine_type, terraform validate will return an error indicating the missing attribute:

terraform validate

Sample output for a successful validation:

Success! The configuration is valid.

Plan

The terraform plan command is used to preview the changes that will be made to your infrastructure based on the current configuration files.

What It Does:

It compares the current state of the infrastructure (recorded in the state file) to the desired state (defined in the configuration files).
It shows you a list of the actions that Terraform will take, such as creating, modifying, or destroying resources.
It provides a dry-run of what will happen without actually making any changes to the infrastructure.

terraform plan -var-file="$TARGET_GCP_PROJECT.tfvars" -out=tfplan

Result:

google_project_service.cloudresourcemanager: Refreshing state... [id=gcp-project-id/cloudresourcemanager.googleapis.com]
google_project_service.iamcredentials: Refreshing state... [id=gcp-project-id/iamcredentials.googleapis.com]
google_project_service.iam: Refreshing state... [id=gcp-project-id/iam.googleapis.com]
google_project_service.sts: Refreshing state... [id=gcp-project-id/sts.googleapis.com]
google_service_account.sa: Refreshing state... [id=projects/gcp-project-id/serviceAccounts/project-sa@gcp-project-id.iam.gserviceaccount.com]
google_storage_bucket.terraform_state: Refreshing state... [id=gcp-project-id-terraform-state]
google_compute_instance.vm_instance-2: Refreshing state... [id=projects/gcp-project-id/zones/asia-southeast1-a/instances/my-vm-instance]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_compute_instance.vm_instance will be created
  + resource "google_compute_instance" "vm_instance" {
      + can_ip_forward       = false
      + cpu_platform         = (known after apply)
      + creation_timestamp   = (known after apply)
      + current_status       = (known after apply)
      + deletion_protection  = false
      + effective_labels     = {
          + "goog-terraform-provisioned" = "true"
        }
      + id                   = (known after apply)
      + instance_id          = (known after apply)
      + label_fingerprint    = (known after apply)
      + machine_type         = "e2-medium"
      + metadata_fingerprint = (known after apply)
      + min_cpu_platform     = (known after apply)
      + name                 = "my-vm-instance"
      + project              = "gcp-project-id"
      + self_link            = (known after apply)
      + tags_fingerprint     = (known after apply)
      + terraform_labels     = {
          + "goog-terraform-provisioned" = "true"
        }
      + zone                 = "asia-southeast1-a"

      + boot_disk {
          + auto_delete                = true
          + device_name                = (known after apply)
          + disk_encryption_key_sha256 = (known after apply)
          + kms_key_self_link          = (known after apply)
          + mode                       = "READ_WRITE"
          + source                     = (known after apply)

          + initialize_params {
              + image                  = "debian-cloud/debian-11"
              + labels                 = (known after apply)
              + provisioned_iops       = (known after apply)
              + provisioned_throughput = (known after apply)
              + resource_policies      = (known after apply)
              + size                   = (known after apply)
              + type                   = (known after apply)
            }
        }

      + confidential_instance_config (known after apply)

      + guest_accelerator (known after apply)

      + network_interface {
          + internal_ipv6_prefix_length = (known after apply)
          + ipv6_access_type            = (known after apply)
          + ipv6_address                = (known after apply)
          + name                        = (known after apply)
          + network                     = "default"
          + network_ip                  = (known after apply)
          + stack_type                  = (known after apply)
          + subnetwork                  = (known after apply)
          + subnetwork_project          = (known after apply)

          + access_config {
              + nat_ip       = (known after apply)
              + network_tier = (known after apply)
            }
        }

      + reservation_affinity (known after apply)

      + scheduling (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

──────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

Apply

Once satisfied with the proposed changes, you use terraform apply to execute the plan. This command:

It executes the changes identified in the terraform plan step.
It updates or creates resources to match the desired state defined in the configuration files.
It modifies the infrastructure while ensuring that the state file is updated accordingly to reflect the new configuration.

For instance, if your configuration includes creating a Google Compute Engine instance with specific parameters, terraform apply will send API requests to Google Cloud to provision the instance according to the configuration.

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Similarly, if you modify an attribute, such as the machine type, Terraform will update the existing resource without recreating it if possible.

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

The combination of plan and apply ensures a safe, iterative process to manage infrastructure changes, giving you visibility and control at every step.

State Management

State management is a critical aspect of Terraform's infrastructure-as-code workflow. Terraform uses a state file to store information about the current state of your infrastructure. This file, typically named terraform.tfstate that acts as a snapshot of the resources managed by Terraform, recording their attributes and configurations. The state file is essential for ensuring that Terraform can manage your infrastructure efficiently and track changes over time.

How State Management Works

Initial State Creation:
When you run terraform apply for the first time, Terraform provisions the resources defined in your configuration and generates a state file. For example, if you create a Google Compute Engine instance, the state file records details such as the instance name, machine type, zone, and IP address.
Tracking and Comparing Changes:
Each time you modify your configuration and run terraform plan or terraform apply, Terraform compares the current state in the state file with the desired state in your configuration. Based on this comparison, Terraform identifies the required actions, such as adding new resources, updating existing ones, or destroying obsolete resources.
State Updates:
After applying changes, Terraform updates the state file to reflect the current state of your infrastructure. This ensures that subsequent operations use accurate and up-to-date information.
State File Locking
Terraform implements state file locking to prevent concurrent operations that could corrupt the state file or lead to inconsistent changes. When using a remote backend (e.g., Google Cloud Storage, AWS S3, or Terraform Cloud), Terraform automatically locks the state file before performing any operations.

For example, if one user is running terraform apply, any other attempts to run Terraform commands that modify the state file will block until the lock is released. Once the operation completes, Terraform unlocks the state file automatically. This mechanism is crucial for teams working collaboratively, ensuring that only one process can modify the state file at a time.

In cases where locking is supported by the backend but fails (e.g., due to misconfiguration), Terraform will notify you with an error message. Manually unlocking a state file should only be done when you're certain no other process is running, as it can lead to inconsistencies.

Key Features of State Management

Resource Dependencies:
The state file tracks resource dependencies, ensuring that Terraform applies changes in the correct order. For instance, it will provision a network before creating resources that depend on it.
Efficient Updates:
Terraform uses the state file to identify and execute only the necessary changes, avoiding resource recreation unless explicitly required.
Remote State Backends:
Storing state files remotely enables better collaboration, secure storage, and features like locking and versioning. Example of configuring a remote backend on Google Cloud Storage:


terraform {
  backend "gcs" {
    bucket = "your-bucket-name"
    prefix = "terraform/state"
  }
}

Sensitive Data in State Files: State files may contain sensitive information such as passwords or API keys. Use encryption and access controls to secure the state file, especially when stored remotely.

Best Practices

Use Remote State Backends: For team environments, always store the state file in a remote backend with locking enabled to avoid conflicts.
Encrypt State Files: Use encryption to protect sensitive data stored in the state file.
Version Control: Check the .terraform.lock.hcl file into version control to ensure consistent provider versions but exclude terraform.tfstate from version control using .gitignore.
Avoid Manual Edits: Never manually edit the state file, as this can lead to inconsistencies and unexpected behavior.
State File Backup: Enable versioning on remote backends to recover previous states if needed.

Modules and Reusability:

In Terraform, modules are a fundamental concept that allows you to organize and reuse infrastructure code. By using modules, you can break down your infrastructure into smaller, reusable components, improving maintainability, scalability, and reducing redundancy. Modules provide a way to group related resources together and treat them as a single unit, making it easier to manage complex infrastructures.

What Are Modules?

A module in Terraform is a collection of resource definitions and configurations that can be reused across different parts of your infrastructure. Modules help you encapsulate resources that serve a specific purpose, such as setting up a web server, creating a database, or configuring networking components. Instead of defining these resources multiple times, you can use a module to reuse the same configuration in different places or projects.

Root Module: The configuration in the directory where you run terraform apply is considered the root module. This is where your main configuration lives.
Child Modules: These are modules that are called by the root module or other modules. They are typically stored in separate directories and referenced in the root module using module blocks.

Structure

Benefits of Using Modules

Reusability: Once a module is written, it can be reused across different projects, environments, or regions. This reduces the need to repeat similar resource definitions and minimizes the risk of errors.
Maintainability: By organizing your infrastructure code into modules, you make it easier to maintain and update. Changes to a module only need to be made in one place and can be propagated throughout all instances where the module is used.
Modularization: Modules enable you to break down complex infrastructure into smaller, more manageable pieces. Each module can focus on a specific task, such as setting up compute resources, networking, or storage, and be reused independently.
Consistency: Modules promote best practices and standardize how resources are configured and managed. This ensures infrastructure is provisioned in a predictable and reliable way.
Collaboration: Teams can share modules across the organization, streamlining the process of setting up common infrastructure components. This encourages collaboration, shared knowledge, and the adoption of infrastructure as code practices.

How to Create and Use Modules

Creating a Module: A module is simply a directory containing Terraform configuration files. For example, you might create a module for a Google Compute Engine instance.

Example of a module to create a Google Compute Engine instance (modules/instance/main.tf):

resource "google_compute_instance" "example" {
  name         = var.name
  machine_type = var.machine_type
  zone         = var.zone

  boot_disk {
    initialize_params {
      image = var.image
    }
  }

  network_interface {
    network       = "default"
    access_config {}
  }
}

(modules/instance/variables.tf)

variable "name" {}
variable "machine_type" {}
variable "zone" {}
variable "image" {}

Using a Module: Once a module is created, it can be called and reused in the root configuration or other modules. You call a module by using the module block, where you specify the source and any necessary input variables.

Example of using the instance module in a root module:

module "web_server" {
  source        = "./modules/instance"
  name          = "web-server-1"
  machine_type  = "e2-medium"
  zone          = "us-west1-a"
  image         = "projects/debian-cloud/global/images/family/debian-11"
}

module "db_server" {
  source        = "./modules/instance"
  name          = "db-server-1"
  machine_type  = "e2-medium"
  zone          = "us-west1-b"
  image         = "projects/debian-cloud/global/images/family/debian-11"
}

Passing Variables to Modules: You can pass variables to a module to customize its behavior for different use cases. For instance, you might pass different machine types, zones, or images when using the same module for different servers.

Example of passing variables:

module "web_server" {
  source        = "./modules/instance"
  name          = "web-server-1"
  machine_type  = "e2-medium"
  zone          = "us-west1-a"
  image         = "projects/debian-cloud/global/images/family/debian-11"
}

Output from Modules: Modules can also define outputs that expose information about the resources they create. This allows you to reference values from a module in other parts of your configuration.

Example of defining an output in a module (modules/instance/outputs.tf):

output "instance_name" {
  value = google_compute_instance.example.name
}

Example of using the output in the root module:

module "web_server" {
  source        = "./modules/instance"
  name          = "web-server-1"
  machine_type  = "e2-medium"
  zone          = "us-west1-a"
  image         = "projects/debian-cloud/global/images/family/debian-11"
}

resource "google_compute_firewall" "allow_http" {
  name    = "allow-http-${**module.web_server.instance_name**}"
  network = "default"

  allow {
    protocol = "tcp"
    ports    = ["80"]
  }
}

Best Practices for Using Modules

Use Clear Naming Conventions: Use descriptive names for both modules and variables to ensure they are easy to understand and reuse.
Version Control for Modules: Store modules in separate directories or even version-controlled repositories to keep track of changes over time.
Use Module Sources: You can source modules from local directories, versioned Git repositories, or the Terraform Registry. The Terraform Registry contains publicly available modules for many common use cases, such as AWS, Google Cloud, and Azure.
Avoid Hardcoding Values: Use variables for values that might change between environments or regions to make modules more flexible and reusable.
Modularize Common Resources: Create modules for commonly used resources (e.g., networking, security groups, VMs) to prevent duplication of configuration across your infrastructure.

The above example illustrates the use of modules from a local file system. Refer to Terraform module sources to learn more about how modules can be passed from other sources too.

## Terraform Destroy: 
The terraform destroy command is used to safely and efficiently delete all resources managed by a Terraform configuration. This command ensures that your infrastructure is removed in a controlled manner, based on the dependency graph that Terraform maintains.

### Key Features of terraform destroy:
**Dependency-Aware Deletion**:
Terraform determines the correct order to delete resources, respecting dependencies between them. For example, it will detach a disk from a compute instance before deleting the instance itself.

**State File Utilization**:
Terraform uses the state file to track all resources under management. It compares the current state with the configuration to identify which resources need to be deleted.

**Selective Resource Deletion**:
While terraform destroy typically removes all resources defined in the configuration, you can use targeted commands to delete specific resources selectively.

### How to Use terraform destroy
Basic Syntax - 
Run the following command in your Terraform workspace to delete all managed resources:

terraform destroy


Terraform will:

1. Load the state file.
2. Generate a plan for deleting resources.
3. Prompt for confirmation before proceeding.

### Options for terraform destroy
**Skip Confirmation**: Add the -auto-approve flag to bypass the confirmation prompt:

terraform destroy -auto-approve

⚠️ Use this cautiously, especially in production environments, as it immediately initiates resource destruction.

**Target Specific Resources**: If you want to delete specific resources only, use the -target flag:

terraform destroy -target=google_compute_instance.example

This command will destroy only the specified resource while leaving others intact.

**Specify a State File**: Use the -state flag to specify a state file if you’re working outside the default location:

terraform destroy -state=custom_state.tfstate


Example output:

google_compute_instance.example: Refreshing state... [id=example-instance]
Plan: 0 to add, 0 to change, 1 to destroy.
Do you really want to destroy? Terraform will delete all resources.
Enter a value: yes

google_compute_instance.example: Destroying... [id=example-instance]
google_compute_instance.example: Destruction complete after 12s.
Destroy complete! Resources: 1 destroyed.


### Additional Considerations
**State File Locking**:
During terraform destroy, Terraform locks the state file to prevent concurrent operations that could cause resource drift or corruption.

**Remote State Management**:
If using a remote backend (e.g., Google Cloud Storage or AWS S3), ensure your state file is accessible and unlocked. Terraform automatically locks and unlocks the remote state during operations.

**Preventing Accidental Deletion**:
Use lifecycle.prevent_destroy in your resource configuration to safeguard critical resources:

resource "google_compute_instance" "example" {
name = "critical-instance"
machine_type = "e2-medium"
zone = "us-west1-a"

lifecycle {
prevent_destroy = true
}
}


**Handling Orphaned Resources**:
If resources were deleted manually or outside of Terraform, terraform destroy might not find them in the state file. Use terraform state rm to clean up the state file.

### Best Practices for Using terraform destroy

- 
Test in Non-Production Environments:
Always test terraform destroy in staging or test environments to understand its impact.

- 
Backup Your State File:
Before running terraform destroy, back up your state file to ensure you can recover from unintended changes.

- 
Dry Run with terraform plan:
Run terraform plan -destroy to preview the destruction plan before executing the actual command:

terraform plan -destroy




In summary, Terraform is a robust tool for automating the creation, deployment, and management of infrastructure using code. Its compatibility with multiple cloud providers and platforms makes it a flexible and powerful choice for infrastructure management.

Infrastructure as Code

sam-nash — Tue, 10 Dec 2024 11:42:19 +0000

In the fast-evolving realm of IT, Infrastructure as Code (IaC) has emerged as a game-changer. This blog aims to explore the nuances of IaC, its importance, benefits, and how it's reshaping the landscape of IT infrastructure management.

What is Infrastructure as Code?

Infrastructure as Code is a practice in IT that involves managing and provisioning computing infrastructure through machine-readable definition files, rather than through physical hardware configuration or interactive configuration tools. In essence, it allows you to treat your infrastructure in the same way you treat your application code.

Why IaC Matters

Consistency and Repeatability: IaC ensures that the same environment is recreated consistently every time. This eliminates the "it works on my machine" problem and provides a uniform environment for development, testing, and production.
Scalability: With IaC, you can easily scale your infrastructure up or down based on demand. This agility is crucial for modern applications that need to handle varying loads.
Speed and Efficiency: Automation is at the heart of IaC. By automating infrastructure provisioning and management, teams can deploy applications faster and more efficiently.
Cost-Effectiveness: Automating infrastructure management reduces the need for manual intervention, which can lead to significant cost savings. Plus, IaC allows for better resource management, ensuring you only pay for what you use.
Version Control: Just like application code, infrastructure code can be version controlled. This means you can track changes, revert to previous versions, and understand the history of your infrastructure configurations.

Key Tools in IaC

Several tools have become synonymous with IaC, each offering unique features and capabilities. Some of the most popular ones include:

Terraform: A versatile tool by HashiCorp, known for its platform-agnostic capabilities and wide community support.
AWS CloudFormation: Amazon's offering, specifically tailored for managing AWS resources.
Ansible: A Red Hat tool that provides simple yet powerful automation for a variety of IT environments.
Puppet: Known for its strong configuration management features and scalability.
Chef: Another powerful configuration management tool, focusing on automation across the entire IT stack.

Getting Started with IaC

Starting with IaC can be a bit overwhelming, but here are some steps to guide you:

Choose the Right Tool: Assess your current infrastructure and choose a tool that best fits your needs.
Learn the Basics: Invest time in understanding the basic concepts and syntax of your chosen tool.
Start Small: Begin with a small project to get hands-on experience.
Integrate with CI/CD Pipelines: Integrate IaC with your Continuous Integration and Continuous Deployment (CI/CD) pipelines for streamlined deployments.
Collaborate and Share: Make use of version control systems like Git to collaborate and share your infrastructure code with your team.

Infrastructure as Code is a paradigm shift in how IT infrastructure is managed. By treating infrastructure like code, organizations can achieve greater consistency, scalability, and efficiency, all while reducing costs and speeding up deployment times.

Google Pub/Sub

sam-nash — Wed, 20 Nov 2024 10:38:43 +0000

Introduction: The Power of Event-Driven Architectures

Have you ever wondered how large-scale systems like Google, Netflix, or Spotify manage to process millions of real-time events, such as sending notifications, updating dashboards, or recommending personalized content, all without missing a beat? The secret lies in event-driven architectures, and at the heart of such systems is a powerful tool: Google Cloud Pub/Sub.

In a world where businesses demand seamless integration and instant processing of data, traditional point-to-point messaging systems often fall short. Enter Google Pub/Sub, a fully managed messaging service designed to decouple applications, handle massive data streams, and enable real-time communication with unparalleled reliability and scalability.

Why Pub/Sub is Essential for Modern Systems
Modern applications require more than just static workflows; they thrive on dynamic, event-driven processes that can adapt and respond to real-time data. Here are some key reasons why Google Pub/Sub has become indispensable:

Real-Time Processing: From IoT devices generating millions of sensor readings to financial systems detecting fraudulent transactions, Pub/Sub ensures these events are captured and processed in real time.
Scalability: Pub/Sub seamlessly scales to handle millions of messages per second, supporting the ever-growing demands of modern businesses.
Decoupling of Services: It simplifies architectures by allowing independent components to communicate asynchronously, improving maintainability and flexibility.
Global Reach: Built on Google's global infrastructure, Pub/Sub offers low-latency message delivery across regions.
Integration-Friendly: It integrates seamlessly with other Google Cloud services like BigQuery, Dataflow, Cloud Functions, and Cloud Storage, making it a cornerstone for building robust, interconnected systems.

Google Cloud Pub/Sub is a fully managed messaging service designed for real-time communication, enabling independent applications to send and receive messages seamlessly. It's a powerful tool for building scalable and reliable systems.

In this tutorial, we'll delve into the core concepts of Pub/Sub, explore practical examples using the command-line interface (CLI) and Python SDK, and discuss various use cases.

Key Concepts

Topic: A named resource to which messages are published.
Subscription: A named resource that receives messages from a specific topic.
Publisher: An application that sends messages to a topic.
Subscriber: An application that receives messages from a subscription.
Message: The core unit of communication in Pub/Sub, consisting of data (payload) and attributes (key-value metadata).
Acknowledgment (Ack): Subscribers must confirm receipt of a message by acknowledging it. Unacknowledged messages are redelivered to ensure reliable processing.
Push vs. Pull Subscriptions
- Pull: Subscribers explicitly fetch messages.
- Push: Pub/Sub pushes messages to an HTTP(S) endpoint.
Dead Letter Queue (DLQ): A separate topic for routing undeliverable messages after a defined number of delivery attempts. Useful for troubleshooting.
Retention Policy: Controls how long messages are stored if they remain unacknowledged or acknowledged. Default: 7 days.
Filters: Enable subscribers to receive only messages that match specific criteria based on attributes.
Ordering Keys: Ensures messages with the same ordering key are delivered in the order they were published.
Exactly Once Delivery: Guarantees no duplicate messages are delivered when configured correctly.
Flow Control: Helps subscribers manage the rate of message delivery to prevent being overwhelmed.
IAM Permissions and Roles: Pub/Sub uses Google Cloud IAM to control access.
- Key roles:
- Publisher: roles/pubsub.publisher
- Subscriber: roles/pubsub.subscriber
- Viewer: roles/pubsub.viewer
Message Deduplication: Prevents duplicate messages caused by retries or network issues.
Message Expiration (TTL): Automatically drops messages from a subscription after a specified time-to-live, reducing storage costs.
Backlog: The collection of unacknowledged messages in a subscription, enabling message replay if required.
Schema Registry: Defines structured message formats (e.g., Avro, Protocol Buffers), ensuring consistent data validation.
Batching: Groups multiple messages together for efficient publishing and delivery, improving performance.
Regional Endpoints: Publish messages to region-specific endpoints for compliance and reduced latency.
Cross-Project Topics and Subscriptions: Share Pub/Sub resources across projects for multi-project architectures.
Monitoring and Metrics: Integrated with Cloud Monitoring, providing insights like throughput, acknowledgment latency, and backlog size.
Snapshot: Captures the state of a subscription at a specific time, enabling message replay from that point.
Message Encryption: Messages are encrypted in transit and at rest by default, with the option to use Customer-Managed Encryption Keys (CMEK).

Getting Started

Pre-requisites

Set Up Your Google Cloud Project:
- Create a new Google Cloud project or select an existing one.
- Enable the Pub/Sub API.
Install the Google Cloud SDK:
- Download and install the SDK from the official website.
- Authenticate your Google Cloud account using
```
 # using cli
 gcloud auth login
```

Creating a Topic and Subscription

# Create a topic using cli
gcloud pubsub topics create my-topic

# Create a subscription using cli
gcloud pubsub subscriptions create my-subscription --topic my-topic

# Using the Python SDK
from google.cloud import pubsub_v1

# Create a Publisher client
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("your-project-id", "my-topic")
publisher.create_topic(topic_path)

# Create a Subscriber client
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path("your-project-id", "my-subscription")
subscriber.create_subscription(subscription_path, topic_path)

Publishing Messages

# using cli
gcloud pubsub topics publish my-topic \
--message "Hello, Pub/Sub!"

# using the Python SDK
# Publish a message
message = "Hello, Pub/Sub!"
future = publisher.publish(topic_path, message.encode("utf-8"))
print(future.result())

Subscribing to Messages

# Using the CLI
gcloud pubsub subscriptions pull my-subscription --auto-ack

# Using the Python SDK
def callback(message):
    print("Received message: {}".format(message.data))
    message.ack()

subscriber.subscribe(subscription_path, callback=callback)

# The subscriber will stay alive until interrupted
while True:
    time.sleep(60)

Use Cases with Practical Examples

Triggers with Cloud Functions

Use Case: Automatically trigger a process whenever a new message is published to a Pub/Sub topic. For example, sending an email notification when a new user registers.

Solution: Use Cloud Functions to listen to Pub/Sub topics and handle messages.

Steps:
Pre-Requisites
-- Enable the apis run.googleapis.com, cloudbuild.googleapis.com, eventarc.googleapis.com
-- Grant the role roles/logging.logWriter to the default compute servie account

gcloud projects add-iam-policy-binding $GCP_PROJECT \
  --member="serviceAccount:<ID>-compute@developer.gserviceaccount.com" \
  --role="roles/logging.logWriter"

Create a Pub/Sub topic:

gcloud pubsub topics create user-registration

Write a Cloud Function:

def send_email_notification(event, context):
    import base64
    message = base64.b64decode(event['data']).decode('utf-8')
print(f"Sending email notification for: {message}")

Deploy the Cloud Function:

  gcloud functions deploy sendEmailNotification \
--runtime python39 \
--trigger-topic user-registration \
--entry-point send_email_notification

Publish a message to test:

  gcloud pubsub topics publish user-registration --message "New User: John Doe"

Result: The Cloud Function logs the email
notification process.

Streaming Data to BigQuery

Use Case: Process and analyze large volumes of event data, such as IoT sensor readings or transaction logs, by streaming messages from Pub/Sub to BigQuery.

Solution: Use a pre-built Dataflow template to ingest data into BigQuery.

Steps:

Create a Pub/Sub topic and subscription:

gcloud pubsub topics create sensor-data
gcloud pubsub subscriptions create sensor-data-sub --topic=sensor-data

Enable BigQuery API and create a BigQuery dataset and table:

# schema.json
[
  {
    "name": "temperature",
    "type": "FLOAT",
    "mode": "NULLABLE"
  },
  {
    "name": "humidity",
    "type": "FLOAT",
    "mode": "NULLABLE"
  }
]

```
bq mk sensor_dataset
bq mk --table sensor_dataset.sensor_table \ 
schema.json
```

Enable the Dataflow API and run the Dataflow job:

# enable dataflow api
https://console.developers.google.com/apis/api/dataflow.googleapis.com/overview?project=your-project-id

```bash
# run the dataflow job
gcloud dataflow jobs run pubsub-to-bigquery \
--gcs-location gs://dataflow-templates-us-central1/latest/PubSub_to_BigQuery \
--region us-central1 \
--parameters inputTopic=projects/your-project-id/topics/sensor-data,outputTableSpec=your-project-id:sensor_dataset.sensor_table
```

Publish messages to the topic:

gcloud pubsub topics publish sensor-data --message '{"temperature": 25, "humidity": 60}'

Result: The message data should appear in the BigQuery table for further analysis.

Logging Events with Sinks

Use Case: Centralize and process logs by routing Cloud Logging events to a Pub/Sub topic for downstream processing, like alerting or archiving.

Solution: Create a logging sink targeting a Pub/Sub topic.

Steps:

Create a Pub/Sub topic:

   gcloud pubsub topics create log-events

Create a logging sink:

    gcloud logging sinks create log-sink \
    pubsub.googleapis.com/projects/your-project-id/topics/log-events

Set IAM permissions for the sink service account:

gcloud pubsub topics add-iam-policy-binding log-events \
    --member=serviceAccount:service-account-email \
    --role=roles/pubsub.publisher

Create a subscriber to process the logs:

from google.cloud import pubsub_v1

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path('your-project-id', 'log-events-sub')

def callback(message):
    print(f"Received log: {message.data.decode('utf-8')}")
    message.ack()

subscriber.subscribe(subscription_path, callback=callback)
print("Listening for log events...")

Result: Log messages flow into Pub/Sub and can be processed by your custom subscriber.

Streaming Messages to Cloud Storage

Use Case: Archive real-time data, such as application usage metrics or user activity logs, to Cloud Storage for long-term storage.

Solution: Use Dataflow to write Pub/Sub messages to a Cloud Storage bucket.

Steps:

Create a Pub/Sub topic and subscription:

gcloud pubsub topics create app-metrics
gcloud pubsub subscriptions create app-metrics-sub --topic=app-metrics

Create a Cloud Storage bucket:

gsutil mb gs://your-bucket-name

Run the Dataflow job:

gcloud dataflow jobs run pubsub-to-gcs \
    --gcs-location gs://dataflow-templates-us-central1/latest/PubSub_to_Cloud_Storage_Text \
    --region us-central1 \
    --parameters inputTopic=projects/your-project-id/topics/app-metrics,outputDirectory=gs://your-bucket-name/data/,outputFilenamePrefix=metrics,outputFileSuffix=.json

Publish messages:

gcloud pubsub topics publish app-metrics --message '{"event": "page_view", "user": "123"}'

Result: Messages are written as JSON files in the specified Cloud Storage bucket.

Conclusion
Google Pub/Sub is a versatile tool for building scalable and reliable applications. By understanding the core concepts and leveraging the CLI and Python SDK, you can effectively utilize Pub/Sub to solve a variety of real-world problems.

Streamline Your GitHub Workflows with Composite Actions

sam-nash — Tue, 19 Nov 2024 07:50:57 +0000

GitHub Actions has become a go-to tool for automating workflows, enabling developers to streamline everything from CI/CD to code quality checks. While it’s powerful out of the box, workflows can quickly become unwieldy as the number of repeated steps grows. Composite actions provide a solution to this problem, enabling you to package reusable steps into modular, maintainable components.

In this blog, we’ll explore what composite actions are, why they’re beneficial, and how to create one using a real-world example.

What Are Composite Actions?

Composite actions allow you to encapsulate multiple steps into a single reusable action. Instead of duplicating code across workflows, you can create a standalone block that performs specific tasks and use it across repositories. Composite actions live within your repository and are defined using YAML, making them easy to version, extend, and customize.

Benefits of using Composite Actions

Here are some reasons to embrace composite actions:

Reusability: Share the same logic across multiple workflows and repositories.
Maintainability: Update logic in one place, and all workflows using it benefit.
Readability: Simplify complex workflows by abstracting repetitive logic.
Modularity: Create building blocks that other developers or teams can easily use.

Example: Terraform Operations as a Composite Action

Let’s dive into a practical example. Suppose you have a GitHub Actions workflow that runs Terraform commands such as init, fmt, validate, and plan for managing infrastructure on Google Cloud. Instead of duplicating these steps across workflows, you can package them into a composite action.

The Problem: Repetition in Workflows

Here’s a snippet of a typical Terraform workflow:

jobs:
terraform:
runs-on: ubuntu-latest
steps:
- name: Terraform Init
run: terraform init -backend-config="bucket=my-project-tfstate"
working-directory: terraform

  - name: Terraform Format
    run: terraform fmt
    working-directory: terraform

  - name: Terraform Validate
    run: terraform

Each step is repeated every time Terraform commands are needed. If you manage multiple workflows or projects, this quickly becomes inefficient and error-prone.

The Solution: A Composite Action

We’ll create a composite action to handle all Terraform operations.

Step 1: Set Up the Directory Structure

Composite actions are stored in the .github/actions/ directory. Create a folder for your action:

.github/actions/terraform/

Inside the folder, create a file named action.yml.

Step 2: Define the Composite Action

Here’s the action.yml file for the Terraform composite action:

name: Terraform Operations
description: Executes Terraform commands like init, fmt, validate, and plan
inputs:
working_directory:
description: Terraform working directory
required: true
gcp_project:
description: Target GCP project
required: true
apply_changes:
description: Whether to apply changes
required: false
default: false
runs:
using: "composite"
steps:
- name: Terraform Init
run: terraform init -backend-config="bucket=${{ inputs.gcp_project }}-tfstate"
working-directory: ${{ inputs.working_directory }}
shell: bash

- name: Terraform Format
  run: terraform fmt
  working-directory: ${{ inputs.working_directory }}
  shell: bash

- name: Terraform Validate
  run: terraform validate
  working-directory: ${{ inputs.working_directory }}
  shell: bash

- name: Terraform Plan
  run: terraform plan -var-file="${{ inputs.gcp_project }}.tfvars" -out=tfplan
  working-directory: ${{ inputs.working_directory }}
  shell: bash

- name: Terraform Apply
  if: ${{ inputs.apply_changes == 'true' }}
  run: terraform apply -auto-approve tfplan
  working-directory: ${{ inputs.working_directory }}
  shell: bash

This composite action supports inputs like the working directory and target GCP project, making it adaptable to different projects.

Using the Composite Action in a Workflow

Once your composite action is defined, you can use it in your workflows like this:

name: Terraform Workflow

on:
push:
branches:
- main

jobs:
terraform:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v2

  - name: Run Terraform Operations
    uses: ./.github/actions/terraform
    with:
      working_directory: terraform
      gcp_project: my-gcp-project
      apply_changes: false

How This Simplifies Your Workflow

• Reduced Duplication: The Terraform logic is centralized in the composite action.
• Improved Readability: The main workflow is concise and easy to follow

Automating Google Cloud Resource Management with GitHub Actions and Workload Identity Federation

sam-nash — Sat, 16 Nov 2024 13:21:47 +0000

In this blog post, we’ll demonstrate how to leverage GitHub Actions (GHA) and Google Workload Identity Federation (WIF) to securely authenticate and create resources on Google Cloud Platform (GCP) using Terraform.

We’ll use two GitHub repositories and two GCloud projects:

Main GitHub Repository:
Contains the Terraform code defining the infrastructure to be provisioned in the Target Project.
Triggers the GitHub Actions workflow upon code changes.

Workflow Repository:
Hosts the GitHub Actions workflow that automates the deployment process.
Authenticates to the Host Project using Workload Identity Federation (WIF).
Executes Terraform commands to apply the infrastructure changes to the Target Project.

Host Google Cloud Project:
Serves as the landing zone for the GitHub Actions workflow.
Configured with WIF, utilises a service account that’s been granted the necessary permissions to access the Target Project.
Does not directly host any infrastructure resources.

Target Google Cloud Project:
Receives the infrastructure changes from the workflow.
Hosts the provisioned resources defined in the Terraform code.

Let’s dive into the setup!

Step 1: Setting Up Google Workload Identity Federation
1.1 - On your host project create a Google Cloud Service Account
Navigate to the Google Cloud Console.
Go to IAM & Admin > Service Accounts.
Create a new service account with the necessary roles for managing your GCP resources.
1.2 - Configure Workload Identity Federation
Go to IAM & Admin > Workload Identity Federation.
Create a Workload Identity Pool and a Provider linked to your GitHub repository.
Follow Google’s official WIF setup guide for detailed instructions. More in a separate Blog Post soon.

Step 2: Permissions
2.1 - Ensure that the target gcp project has a terraform state bucket (ex: project_id-tfstate)
2.2 - Assign relevant roles(roles/editor &
roles/storage.admin) to the Service Account of the Host Project to be able to create/edit resources on the Target Google Project

  gcloud projects add-iam-policy-binding gcp_project_name \
    --member="serviceAccount:service-acct-name@host_gcp_project.iam.gserviceaccount.com" \
    --role="roles/editor"

  gcloud projects add-iam-policy-binding gcp_project_name \
    --member="serviceAccount:service-acct-name@host_gcp_project.iam.gserviceaccount.com" \
    --role="roles/storage.admin"

Step 3: Configure the Main Repo
This repository (e.g., org_name/google_cloud) will contain your Terraform code to manage GCP resources.

Example main.tf:

provider "google" {
  project = var.project
  region  = var.region
}

resource "google_storage_bucket" "bucket" {
  name     = "${var.project}-bucket"
  location = var.region
}

Step 4: Add a Workflow to the Main Repo

Create a GitHub Actions workflow (e.g., .github/workflows/dispatch.yml) in the main repository. This workflow is triggered by events such as push, pull request, or tag creation. When triggered, it makes a GitHub API call to initiate a workflow in another repository (referred to as the Workflow Repo).

Step 5: Configure the Workflow Repo

In the Workflow Repository (Workflow Repo), set up a GitHub Actions workflow dedicated to running Terraform-specific tasks, such as generating a Terraform plan and applying it to the infrastructure. For example, you can name this file .github/workflows/terraform_plan_apply_gcp.yml. Let’s refer to it as the Terraform Workflow.

How It Works

1.Triggering Event:
This Terraform Workflow is designed to be triggered by repository_dispatch events. These events are custom webhook events sent by the main repository via a GitHub API call. When a repository_dispatch event occurs with a specific type, it starts this workflow.
2.Event Types:
You can define specific event types that the workflow should listen to. In this example, the workflow listens for two custom event types:
• terraform_plan: Used to trigger the workflow for generating a Terraform plan.
• terraform_apply: Used to trigger the workflow for applying the Terraform changes.
3.Workflow Configuration:
Add the following YAML configuration to define the trigger mechanism in the Terraform Workflow:

on:
  repository_dispatch:
    types: [terraform_plan, terraform_apply]

• The on: repository_dispatch block specifies that this workflow listens for repository_dispatch events.
• The types field restricts the workflow to respond only to the specified event types (terraform_plan and terraform_apply).

4.Integration with Main Repository:
In the main repository, another workflow (e.g., .github/workflows/dispatch.yml) sends these repository_dispatch events using a GitHub API call. The event payload includes the event type (e.g., terraform_plan) and any additional data required by the Terraform Workflow.
5.Example Workflow Behavior:
• If the main repository triggers a repository_dispatch event with the type terraform_plan, the Terraform Workflow starts and runs tasks related to generating a Terraform plan.
• Similarly, if the event type is terraform_apply, the workflow executes the Terraform apply process to update the infrastructure.

This mechanism decouples the triggering mechanism in the main repository from the Terraform-specific operations in the Workflow Repo, enabling modular and maintainable workflows.

Scenarios to demonstrate how the Main Repo Workflow Operates
Scenario 1: Push with a Tag
A developer pushes Terraform code to a feature branch and adds a tag (e.g., gcp_project_TFPLAN_01).
The dispatch.yml workflow triggers when a tag matching a specific pattern is pushed (e.g., '[a-z]+-[a-z]+TFPLAN[0-9]+').
The workflow determines the Terraform action (e.g., plan) and triggers the Workflow Repo's Terraform workflow via a GitHub API repository_dispatch event.

Scenario 2: Pull Request Creation
When a pull request is opened from a feature branch to develop, the workflow sends a repository_dispatch event with details such as:
event_type: terraform_plan
GCP project name & PR metadata (e.g., PR number, status=opened, merged=false).

Scenario 3: Pull Request Merge
When a pull request is merged, the workflow sends a repository_dispatch event with:
event_type: terraform_apply
GCP project name & PR metadata (e.g., PR number, status=closed, merged=true).

Full workflow

# This GitHub Actions workflow is designed to trigger a Terraform workflow based on specific events.

name: Trigger Terraform Workflow

on:
  push:
    tags:
      - '[a-z]+-[a-z]+_PLAN_[0-9]+'
    # branches:
    #   - 'feature/*'
  pull_request:
    types: [opened, synchronize, closed]
    branches:
      - develop

jobs:
  trigger:
    runs-on: ubuntu-latest
    if: >
      github.event_name == 'push' || 
      (github.event_name == 'pull_request' && 
        (github.event.action == 'opened' || 
          github.event.action == 'synchronize' || 
          (github.event.action == 'closed' && github.event.pull_request.merged == true)))

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      # Debug step to print the GITHUB_REF
      - name: Retrieve GitHub Data
        if: github.event_name == 'push' || github.event_name == 'pull_request' 
        run: |
          echo "GITHUB_REF=${GITHUB_REF}"
          TAG_NAME=${GITHUB_REF#refs/tags/}
          echo "TAG_NAME=$TAG_NAME" >> $GITHUB_ENV

      - name: Print PR Information
        if: github.event_name == 'pull_request'
        run: |
          # Get the latest TAG Name from the source branch
          git fetch --tags

          # Get latest tag
          TAG_NAME=$(git tag --sort=-creatordate | head -n 1)

          # Output the tag
          if [ -z "$TAG_NAME" ]; then
            echo "No tags found on the source branch: $SOURCE_BRANCH"
          else
            echo "The latest tag on the source branch ($SOURCE_BRANCH) is: $TAG_NAME"
            echo "TAG_NAME=$TAG_NAME" >> $GITHUB_ENV
          fi
          echo "PR Number: ${{ github.event.pull_request.number }}"
          echo "PR Action: ${{ github.event.action }}"
          echo "PR Merged: ${{ github.event.pull_request.merged }}"

      - name: Extract Information from Tag or PR
        id: extract_info
        run: |
          GCP_PROJECT=$(echo $TAG_NAME | cut -d'_' -f1)
          echo "GCP_PROJECT=$GCP_PROJECT" >> $GITHUB_ENV
          echo "The Tag Name is: $TAG_NAME"
          echo "The Target GCP Project is: $GCP_PROJECT"
          if [[ "${{ github.event_name }}" == "push" ]]; then
            ACTION="plan"
          elif [[ "${{ github.event_name }}" == "pull_request" ]]; then
            if [[ "${{ github.event.action }}" == "closed" && "${{ github.event.pull_request.merged }}" == "true" ]]; then
              ACTION="apply"
            else
              ACTION="plan"
            fi
          fi
          echo "ACTION=$ACTION" >> $GITHUB_ENV
          echo "The Terraform Action is: $ACTION"

      - name: Trigger Terraform Workflow for Push Commits
        if: github.event_name == 'push'
        run: |
          echo "This was triggered as a result of the Event: ${{ github.event_name }} commits"
          curl -X POST \
            -H "Accept: application/vnd.github.v3+json" \
            -H "Authorization: Bearer ${{ secrets.GH_DISPATCH_PAT }}" \
            https://api.github.com/repos/${{ github.repository }}/dispatches \
            -d "{\"event_type\":\"terraform_${{ env.ACTION }}\", \"client_payload\": {\"repository\": \"${{ github.repository }}\", \"project_name\": \"${{ env.GCP_PROJECT }}\", \"tag_name\": \"${{ env.TAG_NAME }}\"}}"

      - name: Trigger Terraform Workflow for PR
        if: github.event_name == 'pull_request'
        run: |
          echo "This was triggered as a result of PR Number: ${{ github.event.pull_request.number }} being ${{ github.event.action }}"
          curl -X POST \
            -H "Accept: application/vnd.github.v3+json" \
            -H "Authorization: Bearer ${{ secrets.GH_DISPATCH_PAT }}" \
            https://api.github.com/repos/${{ github.repository }}/dispatches \
            -d "{\"event_type\":\"terraform_${{ env.ACTION }}\", \"client_payload\": {\"repository\": \"${{ github.repository }}\", \"pr_number\": \"${{ github.event.pull_request.number }}\", \"pr_event\": \"${{ github.event.action }}\", \"pr_merged\": \"${{ github.event.pull_request.merged  }}\", \"project_name\": \"${{ env.GCP_PROJECT }}\", \"action\": \"${{ env.ACTION }}\"}}"

How the Terraform Workflow Operates
This workflow is hosted in the Workflow Repo and is triggered by repository dispatch events. Below, we break down each step with detailed explanations and corresponding code.

Step 1: Define Workflow Triggers
The workflow listens for certain event types:

repository_dispatch: Triggered by the Main Repo for Terraform plan and apply actions. More on repository dispatch can be found on the Official GitHub Documentation.

name: Terraform CI/CD

on:
  workflow_dispatch:
  repository_dispatch:
    types: [terraform_plan, terraform_apply]

Step 2: Set Up the Terraform Job
Define a Terraform job that runs on ubuntu-latest with relevant permissions for GitHub Actions to interact with the repository and pull requests.

jobs:
  terraform:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
      pull-requests: write
      repository-projects: write

Step 3: Set Environment Variables
Capture the payload data from the triggering event(the main workflow) and set them as environment variables for subsequent steps.

- name: Set ENV variables
  run: |
    echo "TARGET_GCP_PROJECT=${{ github.event.client_payload.project_name }}" >> $GITHUB_ENV
    echo "CLOUDSDK_CORE_PROJECT=${{ github.event.client_payload.project_name }}" >> $GITHUB_ENV
    echo "GH_REPOSITORY=${{ github.event.client_payload.repository }}" >> $GITHUB_ENV
    GH_REPO_NAME=$(echo "${{ github.event.client_payload.repository }}" | cut -d'/' -f2)
    echo "GH_REPO_NAME=${GH_REPO_NAME}" >> $GITHUB_ENV       
    echo "GH_PR_NUMBER=${{ github.event.client_payload.pr_number }}" >> $GITHUB_ENV
    echo "GH_PR_EVENT=${{ github.event.client_payload.pr_event }}" >> $GITHUB_ENV
    echo "GH_PR_MERGED=${{ github.event.client_payload.pr_merged }}" >> $GITHUB_ENV

Step 4: Checkout the Code
Clone the Main Repo containing the Terraform code.

- name: Checkout code
  uses: actions/checkout@v2
  with:
    repository: ${{ env.GH_REPOSITORY }}
    token: ${{ secrets.GITHUB_TOKEN }}

Step 5: Set Up Terraform
Set up Terraform to run commands such as init, plan, and apply.

- name: Set up Terraform
  uses: hashicorp/setup-terraform@v3

Step 6: Authenticate to Google Cloud
Use Workload Identity Federation to securely authenticate with Google Cloud.

- name: Authenticate to Google Cloud
  id: authenticate
  uses: google-github-actions/auth@v2
  with:
    create_credentials_file: true
    workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
    service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}

Step 7: Initialize Terraform
Set the GCP project and initialize Terraform with the remote backend configuration.

- name: Terraform Init
  id: init
  run: terraform init -backend-config="bucket=$TARGET_GCP_PROJECT-tfstate"
  working-directory: ${{ env.TF_WORKING_DIR }}

Step 8: Validate Terraform Code
Ensure the Terraform configuration is correctly formatted and syntactically valid.

- name: Terraform Format
  id: fmt
  run: terraform fmt
  working-directory: ${{ env.TF_WORKING_DIR }}

- name: Terraform Validate
  id: validate
  run: terraform validate
  working-directory: ${{ env.TF_WORKING_DIR }}

Step 9: Generate Terraform Plan
Generate a plan and output the details for review.

- name: Terraform Plan
  id: plan
  run: terraform plan -var-file="$TARGET_GCP_PROJECT.tfvars" -out=tfplan
  working-directory: ${{ env.TF_WORKING_DIR }}

- run: terraform show -no-color tfplan
  id: show
  working-directory: ${{ env.TF_WORKING_DIR }}
## We will use the output of terraform show to write the plan as a comment to the pull request

Step 10: Comment on Pull Requests
If triggered by a pull request, post the plan as a comment for review.

- name: PR Comment
  uses: actions/github-script@v7
  if: github.event.action == 'terraform_plan' && ( env.GH_PR_EVENT == 'opened' || env.GH_PR_EVENT == 'synchronize' )
  env:
    PLAN: "terraform\n${{ steps.show.outputs.stdout }}"
  with:
    github-token: ${{ secrets.GH_PAT }}
    script: |
      const { data: comments } = await github.rest.issues.listComments({
        owner: context.repo.owner,
        repo: process.env.GH_REPO_NAME,
        issue_number: process.env.GH_PR_NUMBER,
      })
      const botComment = comments.find(comment => comment.body.includes('Terraform Format and Style'))
      const output = `#### Terraform Plan\n\`\`\`\n${process.env.PLAN}\n\`\`\``

      if (botComment) {
        github.rest.issues.updateComment({
          owner: context.repo.owner,
          repo: process.env.GH_REPO_NAME,
          comment_id: botComment.id,
          body: output
        })
      } else {
        github.rest.issues.createComment({
          owner: context.repo.owner,
          repo: process.env.GH_REPO_NAME,
          issue_number: process.env.GH_PR_NUMBER,
          body: output
        })
      }

Step 11: Apply the Terraform Plan
If the event is terraform_apply, apply the plan to create resources.

- name: Terraform Apply
  if: github.event.action == 'terraform_apply'
  id: apply
  run: terraform apply -auto-approve tfplan
  working-directory: ${{ env.TF_WORKING_DIR }}

By integrating GitHub Actions and Google Workload Identity Federation, you can establish a secure, automated CI/CD pipeline for managing GCP resources using Terraform. This approach ensures that Terraform plans are reviewed, validated, and applied only after thorough approval, enhancing both security and operational efficiency.

BDD Style API Tests using Cypress and Cucumber

sam-nash — Thu, 23 Feb 2023 00:28:36 +0000

I am just here to beat Medium(the formatting sucks, you cant read more than 3 stories...).

I like BDD(It has a great syntax which anybody can understand). Especially testers, business users, product team who aren't technical.

But BDD shouldn't be used to automate end users' acceptance testing. The whole idea of acceptance testing is to let the customer(end user) look at, feel and use the application like they normally do in a real world, but not automate their behaviour.

BDD might be useful for Product Owners/Business Analysts, but again, is it worth the effort of a automation engineer to implement BDD for those 1 or 2 people?

And especially for API Testing, I firmly believe BDD is a NO. I think a PO or BA can see the developer or tester demo it to them in the Sprint review.

BDD is GOOD but I think it is an overkill for API Testing(Or any backend tests). But we'll still explore how to do that today.

Recently, I have become a great fan of cypress.io and started to use it for almost everything.

Cypress is an open-source end-to-end testing framework used to test web applications. It provides a fast, reliable, and easy-to-use testing experience, with an intuitive API and an extensive set of built-in commands.
Cypress is great. In the last 1 year there have been 3 major version upgrades, easy to use, open source, community support is increasing.

Cucumber is a popular tool used for behavior-driven development (BDD). It allows you to define test cases in a human-readable format, which can be easily understood by non-technical stakeholders. By using Cucumber with Cypress, you can create tests that are easy to write, easy to read, and easy to maintain.

Overview of the Git Repo

The git repo contains a sample project that demonstrates how to use Cypress with Cucumber.

The tests are organized into feature files, which are meant to describe different aspects of the weather api. For example, the stations.feature file contains test cases related to the functionality of weather station creation, & we could add another solarRadion.feature file that can contain test cases related to the Solar Radiation API.

Each feature file contains one or more scenarios, which describe a specific test case. Scenarios are written in Gherkin syntax, which is a human-readable format that allows you to describe the expected behavior of the web application.

Our API under Test

We'll make use of the OpenWeatherMap Stations API for our tests.

The API exposes the following methods :

POST(to register a stations)
PUT(to update the information about a station)
GET(to retrieve all the stations or a specific station using the stationId)
DELETE(a station that was created by the current user)

Generate an API Key

The Open Weather uses an API Key for Authentication.
To generate and use the apiKey in your tests :
Register online at OpenWeather and generate an api key.

Set Up and Install

If you want to skip all the step-by-step installations and set up, then I recommend that you clone this repo.
Once cloned, cd to the cloned directory and perform npm install
Browse through some of the tests that I wrote already to perform POST + GET operations and to verify the response.

Cypress configuration

The default cypress configuration cypress.config.js has the basics that we need to test the API + Reporting.
Additionally create a cypress.env.json file at the root of your project folder & update the file like below. Replace apiKey with the key that was generated previously.

example :
{ "appid": "<apiKey>" }

NOTE : Let this cypress.env.json file be local to your machine & gitignored. It is a best practice to generate this dynamically or store this as an env var in the CI server.

Cypress custom commands

As we might make use of cy.request() multiple times with different methods, I've created custom commands in (commands.js)[cypress/support/commands.js]
This makes it user friendly(for users who don't want to be too bothered with the technical details of how to make requests).
Better readability of code.

To Run the tests

From your proj dir: type this command on the console and hit return.
npm run cucumberTest

You should see the tests running and pass. You should also see the test results printed on the console.

If you'd like to see a html report, navigate to cypress/reports and open the index.html

Under the Hood

So what actually happened here ?

The Tests rely on two files - the Feature and the spec or implementation

The Feature file is the file which contains the Feature & the associated scenarios that are being tested in the Given When Then And format.

The Spec file is the implementation of the feature file. This is where Cypress code resides that makes API calls and uses assertions to verify the response received.

The Given, When, Then & And sections in the Feature file have matching Given, When, Then & And code blocks in the Spec file.

Feature = 'saying'
Spec = 'doing'

Construct - Feature/Test Organization

a. All Feature files must be created with the file name convention name.feature under the cypress/e2e folder [example: stations.feature]
b. Create a corresponding folder under e2e that has the same name as the feature name above(without '.feature') [example : stations]
c. Create/add your cypress spec/feature implementation under this folder [example : /e2e/stations/apiTest.cy.js]
d. To add your own scenarios and tests, edit the feature file and edit/add more API test scenarios using the Given, When, Then & And construct.
e. And then add the matching tests to the apiTest.js

Approaches

To demonstrate API Testing with Cypress & Cucumber, I have added tests that make use of two approaches as described below.

DATATABLE - Feature is in the stations.feature file that has the test data in a tabular form(to test multiple data combos) & the cypress spec is in the e2e/stations/apiTest.js which has the implementation. The cypress spec loops through the data to POST and performs a response code verification & stores each stationId in an array. It then also sends a GET request for each stationId & verifies the response to check if it matches the original request that was previously sent in the POST call.

For any Scenario :
GIVEN

In the Given section of the feature file we pass a table as the data.

Given I have the following station data to post:
            | external_id  | name                       | latitude | longitude | altitude |
            | DEMO_TEST001 | Team Demo Test Station 001 | 33.33    | -122.43   | 222      |
            | DEMO_TEST002 | Team Demo Test Station 002 | 44.44    | -122.44   | 111      |

In the Given section of the spec(implementation) file, We pass the above datatable as a parameter & then make use of the Cucumber method: hashes() that belongs to the class: dataTable to return an array of objects & then convert this datatable into a Map.

Given('I have the following station data to post:', (dataTable) => {
  // Convert the data table to an array of objects with header keys
  requestData = dataTable.hashes().map((row)

We then convert string representation of numbers and floats to their respective types. If we dont do this the numbers and float values will be converted to strings.(We cant send latitiude as a string to the API).

    Object.keys(row).forEach((key) => {
      const value = row[key];
      if (!isNaN(value)) {
        if (value.includes('.')) {
          row[key] = parseFloat(value);
        } else {
          row[key] = parseInt(value, 10);
        }
      }
    });

Then we wrap this array of objects as an alias for a later use.

cy.wrap(requestData).as('requestData');

WHEN

In the feature file we specify the expected outcome of the POST call by specifying the http status code as a number.

When I post the station data request body to the Create Stations API, then I receive a response status code 201 and an unique alphanumeric stationId in the response

In the spec file, we make a POST call using the cypress command we developed for this(the command basically sends a cy.request() with the required parameters like method, requestbody, apiKey).

When(
  'I post the station data request body to the Create Stations API, then I receive a response status code {int} and an unique alphanumeric stationId in the response',
  (statusCode) => {
    cy.get('@requestData').each((request) => {
      cy.createStation(apiKey, request).then((response) => {
        //create an alias that stores the response received from the api.
        expect(response.status).to.eq(statusCode); //verify the http status code
        expect(response.body.ID).to.match(/^[a-z0-9]+$/i);
        //store the stationId for later use to retrive the station for verification
        stationIds.push(response.body.ID);
      });
    });
  }
);

We then perform assertions on the statusCode that was passed as parameter to When.
Note that we passed the actual status code as 201 in the feature but in the implementation, we specified it as {int}. This is to tell the spec what to read from this position from the feature file.
We then store the stationIds for a later use to make the GET call.

THEN

Finally, in the Then section we make another API GET call for each station that was previously extracted and stored in an array & perform the required assertions.

Then(
  'When the unique stationId is queried using a GET request, the api returns a {int} status code',
  (statusCode) => {
    //Call the GET API with the stationId created using the POST Request & store its response in an alias
    stationIds.forEach((stationId) => {
      cy.getStation(stationId).then((response) => {
        expect(response.status).to.eq(statusCode);

Without DATA TABLE - Feature is in the apiTests.feature & its implementation in the e2e/apiTests/apiTests.js. Here the scenarios have been separated for each POST Request(1 per each test data) and corresponding GET requests verify the data.

The ONLY intention to separate it this way is to demonstrate that an api request body can be added in the scenario's Given or When.
When it comes to reporting, in the former approach, the test report displays the various tests(for each test data combo) under the same scenario and in the latter, the report displays the various tests/scenarios separately.

Happy Testing!