DEV Community: Manish R Warang

TOON: The Token Ninja

Manish R Warang — Fri, 21 Nov 2025 05:22:09 +0000

Because your LLM doesnt need another 4,000-character JSON blob to cry about.

The Scene Every Cloud Architect Knows Too Well

Its Friday evening. Youve just wrapped up a solid week of architecture reviews and CI/CD firefighting.

Then suddenlyPagerDuty pings.

Your AI-powered deployment validator has failed again.

The LLM was supposed to summarise an Azure App Service deployment manifest.

Instead, it choked on a 19KB JSON blob and replied:

I cannot process this PowerPoint file.

You check the logs.

The JSON blob fed into the LLM has:

42 services
118 policy rules
7 nested objects spelling doom
And a token bill that could fund a small island nation

Meanwhile, your FinOps team messages you:

Why did one prompt consume 18,432 tokens?!

Just as you stare into the void, contemplating a career in the tea-making business

TOON walks in quietly.

Like a minimalist samurai.

Ready to solve your token trauma.

Meet TOONToken-Oriented Object Notation

A compact, human-readable, LLM-friendly format designed for the AI era.

TOON is what happens when JSON goes to coding bootcamp, takes minimalism seriously, and stops hoarding punctuation.

Its designed to serialise structured data in a way that LLMs understand more easilyusing fewer tokens and with fewer hallucinations.

If JSON is a 20-page resume,

TOON is the clean 1-page CV that actually gets read.

Why TOON Matters (Especially If You Live in the Cloud)

LLMs struggle with noisy punctuation, long prompts, deeply nested brackets, and repeated field names.

TOON fixes that by:

Removing unnecessary syntax
Flattening repetitive structures
Compressing arrays into tabular form
Reducing token count by 3060%

For engineers dealing with:

API specs
Kubernetes manifests
Azure Bicep outputs
CI/CD summaries
Observability logs

TOON becomes an instant superpower.

Few Cloud & DevOps Examples

ExampleKubernetes Pod Health Summary

JSON

{

"namespace": "prod",

"pods": [

{ "name": "web-7f984d", "status": "Running", "restarts": 1 },

{ "name": "payments-4f29d1", "status": "CrashLoopBackOff", "restarts": 5 }

]

}

TOON

namespace: prod

pods[2]{name,status,restarts}:

web-7f984d,Running,1

payments-4f29d1,CrashLoopBackOff,5

Example 2 Observability Metrics (from official TOON benchmarks)

JSON (5 rows)

{

"metrics": [

{

"date": "2025-01-01",

"views": 5715,

"clicks": 211,

"conversions": 28,

"revenue": 7976.46,

"bounceRate": 0.47

}

]

}

TOON

metrics[5]{date,views,clicks,conversions,revenue,bounceRate}:

2025-01-01,5715,211,28,7976.46,0.47

...

Perfect for telemetry, Grafana streams, log analysis, and dashboards.

When Should You Use TOON? (Straight from the Spec)

Use TOON When:

Your data is mostly flat or semi-flat
Youre sending structured data to an LLM
You want to reduce token costs
You need to handle large arrays of similar objects
Youre feeding logs, metrics, or infra configs to an AI agent

Avoid TOON When:

Your structure is deeply nested (complex trees, recursive objects)
You need strong schema validation
Youre interacting with APIs expecting strict JSON
Your data has highly irregular shapes

A good rule:

If JSON looks like a mazekeep JSON.

If JSON looks like a spreadsheetuse TOON.

TOON vs JSON vs YAMLDeveloper Cage Match

Installation & Quick Start (Developers Love This Part)

CLI (no installation required)

npx @toon-format/cli input.json -o output.toon

Pipe from stdin

echo '{"name":"Ada"}' | npx @toon-format/cli

Library (TypeScript)

npm install @toon-format/toon

Usage

import { encode } from '@toon-format/toon';

console.log(encode({

users: [

{id: 1, name: 'Alice'},

{id: 2, name: 'Bob'}

]

}));

Final ThoughtsThe Future Is TOON-Shaped

TOON isnt just another data format.

Its a practical upgrade for anyone using LLMs in Cloud or DevOps.

It reduces token costs.

It improves AI accuracy.

It removes JSON noise.

It cleans up your prompts.

And most importantly

It gives your AI agents a fighting chance to understand your infrastructure.

Try converting one of your ugly JSON prompts today.

Your LLMand your token billwill thank you.

]]>

Secure by Design: Integrating Security Policies from Code to Cloud with Terraform

Manish R Warang — Fri, 17 Jan 2025 04:03:59 +0000

Introduction: A Balancing Act Between Agility and Security

In the fast-paced world of Cloud computing, security has often been treated as an afterthought, a necessary evil tacked on after infrastructure is already deployed. But with Cloud-native architectures growing more complex by the day, this approach simply doesn’t cut it anymore. We can no longer afford to rely on traditional security practices to safeguard our sprawling infrastructure. Instead, what if we could bake security directly into the infrastructure provisioning process? Welcome to the era of “Secure by Design”, where Terraform allows us to enforce security policies at every stage — from code development to Cloud deployment.

Section 1: The Evolution of Cloud Security and Infrastructure as Code (IaC)

Back in the days when on-premise servers roamed the Earth, security was a well-guarded perimeter. Firewalls, intrusion detection systems, and manual audits dominated the scene. But as we transitioned to Cloud-native environments, security became everyone’s responsibility — or in other words, nobody’s problem.

Infrastructure as Code (IaC) tools like Terraform revolutionized the way we manage infrastructure, but it also introduced new challenges. Mismanaged credentials? Publicly exposed S3 buckets? Misconfigured access policies? If you’ve spent enough time in Cloud environments, you’ve seen it all. It’s no wonder that IaC without security by design often results in unpredictable configurations that might look like a security nightmare.

Luckily, Terraform is not just about infrastructure automation; it’s about integrating security as an intrinsic part of that automation. It helps ensure that your Cloud infrastructure is deployed securely, efficiently, and most importantly, consistently.

Section 2: Terraform’s Core Principles: The Foundation of Secure Infrastructure

Before diving deeper into security features, let’s understand how Terraform’s fundamental principles naturally align with, and enhance security practices. These principles aren’t just architectural choices — they’re the bedrock upon which we build secure infrastructure.

Declarative Configuration

The declarative nature of Terraform isn’t just about simplicity — it’s your first line of defence. By declaring the desired state rather than writing procedural steps, you reduce the risk of configuration drift and security misconfigurations. When your infrastructure is defined as code, security becomes visible, reviewable, and most importantly, reproducible.

Security Impact : Every resource’s security configuration is explicitly declared, making it impossible to “accidentally” skip security settings. For instance:

resource "aws_s3_bucket" "data_bucket" {
  bucket = "secure-company-data"
  versioning {
    enabled = true # Explicitly declared security feature
  }
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256" # Explicit encryption requirement
      }
    }
  }
}

State Management

Terraform’s state management isn’t just about tracking resources — it’s about maintaining security consistency. The state file contains your infrastructure’s entire security posture, making it a crucial security artifact that needs protection.

Best Practice : Always use remote state storage with encryption and proper access controls:

terraform {
  backend "s3" {
    bucket = "terraform-state-secure"
    key = "prod/terraform.tfstate"
    region = "us-west-2"
    encrypt = true
    kms_key_id = "arn:aws:kms:us-west-2:111122223333:key/1234abcd…"
  }
}

Resource Graph

Terraform’s resource graph isn’t just for dependency management — it’s a security relationship mapper. Understanding these relationships is crucial for security, because it helps identify potential security implications of resource changes.

Security Application : Use the graph to visualize security group dependencies and ensure proper network segmentation:

# Database security group depends on application security group
resource "aws_security_group_rule" "db_ingress" {
  security_group_id = aws_security_group.database.id
  type = "ingress"
  from_port = 5432
  to_port = 5432
  protocol = "tcp"
  source_security_group_id = aws_security_group.application.id
}

Provider Architecture

Terraform’s provider architecture isn’t just about multi-Cloud support — it’s about standardizing security across different platforms. Each provider enforces platform-specific security best practices while maintaining a consistent security approach.

Implementation example :

provider "aws" {
  region = "us-west-2"
  assume_role {
    role_arn = "arn:aws:iam::ACCOUNT_ID:role/TerraformExecutionRole"
    session_name = "TerraformSecureSession"
  }
  default_tags {
    tags = {
      Environment = "Production"
      SecurityLevel = "High"
      DataClassification = "Confidential"
    }
  }
}

Section 3: Key Security Features in Terraform — And Why You Should Care

Terraform, despite its innocent-looking HCL syntax, is the unsung hero of secure Cloud infrastructures. But let’s not just take its word for it. Let’s break down the key security features Terraform offers and see how they directly translate into real-world use cases.

Provider Authentication and Authorization : Terraform ensures secure communication with Cloud providers like AWS, Azure and Google Cloud through provider-specific authentication methods. Imagine managing multiple Cloud providers — Terraform helps you authenticate securely without juggling credentials like hot potatoes.
Managing Secrets and Sensitive Data : Secrets management is the Achilles’ heel of infrastructure security. Storing sensitive data like API keys in plain text is like handing over your house keys to a stranger. With Terraform, you can integrate tools like HashiCorp Vault or AWS Secrets Manager to keep those secrets, well, secret.

Use Case : In a multi-Cloud environment, where sensitive credentials need to be shared across platforms, Terraform can abstract these credentials securely, so they are never exposed, ensuring compliance with security standards.

Remote State Storage and Encryption : Storing your Terraform state remotely, especially for larger teams, is non-negotiable. Using Terraform’s remote state with encryption ensures that your state files (which hold the keys to your infrastructure) are both available and secure.

Use Case : Consider an enterprise spanning different geographies — remote state storage backed by strong encryption ensures that your infrastructure’s blueprint is safe, even when collaborating across regions.

IAM and Access Policies : Following the principle of least privilege, Terraform allows you to manage IAM policies across various Cloud providers. Over-provisioning access is a rookie mistake, and Terraform’s fine-grained control makes sure that your users only get the access they deserve.

Use Case : For a financial services company managing multiple AWS accounts, IAM policies controlled via Terraform ensure that users and services only have access to what they truly need, reducing the risk of a breach.

Section 4: Integrating Security Policies into Terraform Workflows — Because Automation Without Security is a Liability

So now that we’ve established that Terraform can handle security, how do we automate it? Enter Policy as Code , where security policies are encoded into your Terraform workflows to ensure that nothing slips through the cracks. Tools like HashiCorp Sentinel and Open Policy Agent (OPA) enable policy enforcement as part of your infrastructure’s lifecycle.

Example 1: Enforcing Encryption for Storage

resource "aws_s3_bucket" "secure_bucket" {
  bucket = "secure-data-storage"
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

In this example, Terraform ensures that your S3 bucket has server-side encryption enabled by default. No more excuses for “Oops, I forgot to turn on encryption!”

Example 2: Restricting Public Access to Resources

resource "aws_security_group" "secure_sg" {
  name = "no_public_access"
  description = "Security group with restricted public access"
  ingress {
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["10.0.0.0/16"] # No public IP allowed
  }
}

This security group example makes sure that only internal IPs from your network can access certain services — effectively shutting the door to the outside world. Let’s see someone misconfigure that!

Section 5: Securing the Entire Lifecycle — From Code to Cloud

Security in Development : Developers often leave security for later. Bad move. By integrating Terraform early in the development phase, they can proactively implement best practices like encryption, access restrictions and logging from day one.

Example 3: Secure CI/CD Pipelines

resource "aws_codebuild_project" "secure_build" {
  name = "secure-ci-pipeline"
  environment {
    compute_type = "BUILD_GENERAL1_SMALL"
    image = "aws/codebuild/standard:4.0"
    environment_variable {
      name = "SECURE_VAR"
      value = "sensitive_value"
      type = "SECRETS_MANAGER"
    }
  }
}

With Terraform, CI/CD pipelines don’t just automate deployments — they enforce security as they go. This pipeline ensures sensitive variables are stored securely, so every deployment step is auditable and safe.

Post-Deployment Monitoring : Once your infrastructure is deployed, the job isn’t done. Terraform integrates with Cloud-native security tools like AWS Security Hub and Azure Security Centre to continuously monitor your environment.

Example 4: Continuous Monitoring with Terraform

resource "aws_securityhub_account" "example" {
  enable_default_standards = true
}

This simple snippet ensures that Security Hub is activated in your AWS account, constantly scanning for security threats. Sleep tight — Terraform’s got your back.

Section 6: Bringing It All Together — Native Principles Meet Security Practices

The magic happens when we combine Terraform’s native principles with security features. This integration creates a robust security framework that’s both powerful and maintainable:

Modular Security : Use Terraform modules to create reusable security configurations that enforce your organization’s standards:

module "secure_vpc" {
  source = "./modules/secure-vpc"
  environment = "prod"
  enable_flow_logs = true
  enable_network_firewall = true
}

Workspaces for Security Isolation : Leverage Terraform workspaces to maintain separate security contexts for different environments:

terraform workspace select prod
# Production-specific security configurations

Data Sources for Security Compliance : Use data sources to query existing security configurations and ensure compliance:

data "aws_security_group" "existing" {
  name = "production-baseline"
}

Conclusion: Security is Not Optional

In the end, security isn’t just a feature — it’s a necessity. By adopting a “Secure by Design” approach with Terraform, you’re not just building infrastructure; you’re building a fortress that evolves with your needs. Don’t wait until it’s too late — start integrating security policies into your Terraform workflows today.

Key Takeaways :

Start enforcing security policies in your Terraform workflows from day one.
Use tools like HashiCorp Sentinel or OPA to automate security checks.
Implement secure CI/CD pipelines to ensure continuous security throughout the deployment lifecycle.
Leverage Cloud-native security tools for continuous post-deployment monitoring.
Embrace Terraform’s native principles as the foundation for secure infrastructure.

Call to Action : Evaluate your current Terraform setup. It’s time to stop playing catch-up with security and start being proactive. Go ahead — give your infrastructure the security makeover it deserves, built on the solid foundation of Terraform’s core principles.

The Terraform Fork in the Road

Manish R Warang — Mon, 23 Sep 2024 09:39:38 +0000

A Multi-Cloud Architect’s Guide to Choosing Your IaC Path

Introduction

As a multi-cloud architect in the industry, I have the privilege of guiding organizations through complex Cloud infrastructure decisions. Today, we’re facing a pivotal moment in the Infrastructure as Code (IaC) landscape, prompted by significant changes involving Terraform and the rise of OpenTofu. This blog aims to provide insights into the ongoing debate between sticking with Terraform or migrating to the open-source alternative, OpenTofu. Given recent events such as IBM’s acquisition of HashiCorp and Terraform’s licensing changes, it’s crucial to assess the implications for your organization. While there’s no one-size-fits-all answer, understanding the nuances will help you make an informed decision that aligns with your unique needs and risk tolerance.

The Terraform Landscape: Before and After

Terraform’s Rise

Terraform has long been the de-facto standard for Infrastructure as Code (IaC). Its declarative configuration language, extensive provider ecosystem, and ability to manage infrastructure across multiple Cloud platforms have made it indispensable for DevOps teams worldwide. Its open-source nature fostered a robust community, contributing to its rapid adoption and evolution. Organizations of all sizes have relied on Terraform for its stability, scalability and extensive documentation.

IBM Acquisition and BSL

The landscape changed dramatically with IBM’s acquisition of HashiCorp and the shift to the Business Source License (BSL) for Terraform. The BSL, while allowing free use under certain conditions, introduces restrictions that could impact enterprises relying heavily on Terraform. This shift has raised questions about the future trajectory of Terraform, particularly concerning potential cost implications, and the prioritization of features that serve IBM’s strategic goals.

Community Response

In response to these changes, the community has rallied around OpenTofu, an open-source fork of Terraform. OpenTofu aims to maintain the open-source ethos that originally made Terraform popular. It promises to be a free and community-driven alternative, addressing concerns about vendor lock-in and licensing costs. This development has sparked a significant debate within the DevOps community about the best path forward.

Key Considerations for Decision-Makers

Cost

OpenTofu : As an open-source project, OpenTofu eliminates licensing costs, which can be a significant factor for organizations, especially those operating at scale. This cost-saving can be redirected towards other strategic initiatives or infrastructure improvements.
Terraform : The introduction of the BSL necessitates a careful evaluation of the potential financial impact. Organizations must assess whether the benefits of staying with Terraform outweigh the costs associated with its new licensing model.

Feature Parity and Roadmap

OpenTofu : It is crucial to assess whether OpenTofu’s current feature set meets your organization’s requirements. While OpenTofu aims to replicate Terraform’s capabilities, it is still in its nascent stage. Evaluating its roadmap and community support is essential to ensure it aligns with your long-term goals.
Terraform : Under IBM’s stewardship, Terraform’s future roadmap may prioritize enterprise features that align with IBM’s strategic objectives. Organizations need to consider whether these priorities align with their own goals, and whether they can trust IBM to continue developing Terraform in a way that meets their needs.

Community and Support

OpenTofu : While OpenTofu benefits from a growing community eager to contribute, it may not yet match Terraform’s maturity in terms of documentation, plugins and support. However, the open-source community has a track record of quickly filling gaps and addressing issues.
Terraform : Terraform’s extensive community, comprehensive documentation and professional support options make it a reliable choice for organizations requiring robust support and resources. This established ecosystem can be crucial for enterprises that need guaranteed reliability and quick resolution of issues.

Risk Tolerance and Vendor Lock-in

OpenTofu : One of the main advantages of OpenTofu is the lower risk of vendor lock-in. Organizations retain full control over their IaC tool, which can be a significant advantage in terms of flexibility and autonomy. However, there are potential concerns about long-term support and development continuity that need to be addressed.
Terraform : Reliance on a proprietary tool owned by a large corporation like IBM introduces certain risks, such as changes in licensing, pricing, or development focus. Organizations need to evaluate their comfort level with these risks and the potential impact on their operations.

Migration Effort

OpenTofu : Designed for compatibility with Terraform, OpenTofu aims to minimize migration complexity. However, the specific effort required will depend on the intricacies of your current infrastructure and how deeply integrated Terraform is within your systems.
Terraform : Staying with Terraform avoids the immediate cost and effort of migration, but requires careful consideration of the long-term implications of the BSL and IBM’s strategic direction.

Recommendations

For Organizations Highly Invested in Terraform

Evaluate the Impact of the BSL : Conduct a thorough analysis of how the BSL affects your organization. Consider both the direct costs and the potential indirect effects on your operations.
Engage with HashiCorp : Reach out to HashiCorp to understand their licensing terms and future plans. This engagement can provide clarity and help you negotiate terms that align with your needs.
Explore OpenTofu as an Alternative : If cost or vendor lock-in are major concerns, start exploring OpenTofu as a potential alternative. Pilot projects can help assess its suitability and ease of migration.

For Organizations Starting New IaC Projects

Compare Features and Community Support : Conduct a detailed comparison of Terraform and OpenTofu in terms of features, community support and ecosystem maturity. This comparison should include both current capabilities and future roadmaps.
Consider Long-Term Costs and Risks : Factor in the long-term cost implications and your organization’s risk appetite. Assess whether the potential savings and flexibility of OpenTofu outweigh the stability and support offered by Terraform.

Conclusion

The decision between Terraform and OpenTofu is nuanced and depends on your organization’s unique circumstances. While Terraform’s established ecosystem and professional support make it a compelling choice, the cost implications of the BSL and potential vendor lock-in under IBM’s ownership are significant considerations. OpenTofu offers a promising open-source alternative that can reduce costs and increase flexibility, but it requires careful evaluation of its maturity and community support.

I encourage you to actively assess your options, engage with both communities, and make an informed choice that aligns with your organization’s goals and values. The landscape of Cloud infrastructure management is rapidly evolving, and staying informed and adaptable is key to navigating these changes successfully.

12 Cloud Commandments: Applying 12 Factor App Principles to Master Terraform — Part 4

Manish R Warang — Mon, 24 Jun 2024 12:47:11 +0000

12 Cloud Commandments: Applying 12 Factor App Principles to Master Terraform — Part 4

Processes

In the previous part of this series, we delved into how the 12 Factor App principles advocate for executing tasks as stateless, isolated processes. This approach helps prevent resource inefficiencies and system failures. By ensuring that background processes remain lightweight, scalable and resilient, we safeguard our infrastructure’s stability and efficiency.

Port Binding

Port binding was discussed as a crucial practice for ensuring efficient traffic distribution and high availability. We highlighted scenarios like port collisions and ephemeral port issues, emphasizing the importance of dynamic port mapping to avoid conflicts and ensure seamless scaling.

Concurrency

Concurrency involves managing multiple processes efficiently to prevent resource collisions. Using strategies like Terraform workspaces and state locking mechanisms helps maintain resource isolation and prevent conflicts, thus ensuring the stability and integrity of the infrastructure.

In the below part, we will continue our exploration of the 12 Factor App principles in the context of Terraform, focusing on additional principles and their practical applications.

Disposability — Maximize robustness with fast start-up and graceful shutdown

It’s the magic trick of your infrastructure show — it’s all about making things disappear without a trace. But beware the magician who forgets to clean up after themselves — keep your disposability tidy and your stage clear for the next act!

Scenario 1: The “Server Zombie Apocalypse”

Imagine you’re managing a fleet of servers in the Cloud, each responsible for a critical component of your application. Suddenly, one of them decides to play dead in the middle of the night, just like a tired toddler refusing to sleep. You try everything to revive it, from CPR (Cloud Provider Resuscitation) to offering it virtual chicken soup, but nothing works. Now you’re stuck in a server zombie apocalypse, where one unresponsive instance threatens to bring down your entire application like a house of cards. It’s like a horror movie where the villain is a stubborn piece of hardware instead of a masked maniac.

Scenario 2: The “Patchwork Quilt of Pain”

Picture this: you’re responsible for maintaining a legacy system that’s been around since the dawn of Cloud computing. Every time you need to apply a security patch or update a component, it’s like performing surgery on a Frankenstein’s monster of code. You’re forced to patch up holes and stitch together workarounds just to keep the system limping along like a wounded gazelle. The result? A patchwork quilt of pain, where every fix introduces new vulnerabilities and instability. It’s like trying to remodel a house made of Jenga blocks, where every change threatens to bring the whole structure crashing down.

Scenario 3: Lack of graceful shutdown handling in Terraform deployments

One common challenge faced by Cloud and DevOps engineers is ensuring the graceful disposal of resources when scaling down or updating infrastructure. Without proper handling, abrupt termination of resources can lead to data loss or service disruption. Let’s take the example of an auto-scaling group in AWS managed by Terraform. When instances are terminated due to scaling down or updating, there might be ongoing processes or connections that need to be gracefully handled to prevent data loss or service interruptions.

resource "aws_autoscaling_group" "example" {
// Other configurations…
lifecycle {
// Graceful handling of instances during scale down
ignore_changes = ["desired_capacity"]
create_before_destroy = true
  }
}

By utilizing Terraform’s lifecycle block, engineers can ensure that instances are gracefully replaced during updates or scaling down, allowing ongoing processes to complete without interruption.

In conclusion, the “Disposability” principle of the Twelve-Factor App is crucial for maintaining resilience and efficiency in Cloud infrastructure managed with Terraform. As Cloud and DevOps engineers, we face the challenge of gracefully disposing of resources during scaling down or updates. Without proper handling, abrupt terminations can lead to data loss or service disruption. By adhering to disposability principles, such as managing connections and processes effectively, we ensure a smoother transition during scaling events. With Terraform, leveraging features like lifecycle hooks and graceful termination policies, we can mitigate risks and enhance the reliability of our infrastructure. Embracing disposability not only fosters resilience, but also paves the way for seamless scaling and maintenance in dynamic Cloud environments.

Dev/Prod Parity — Keep development, staging and production environments as similar as possible

These are the twin towers of your infrastructure skyline — they should look alike, walk alike, and talk alike. But watch out for the evil twin who tries to steal the spotlight — keep your dev and prod environments in sync and your audiences guessing!

Scenario 1: The “Works on My Machine” Conundrum

You’ve spent hours perfecting your code in the cozy confines of your development environment. Everything seems to be running smoothly, and you’re ready to unleash your masterpiece into the wild. But as soon as you push it to production, disaster strikes. Your app crashes, users revolt, and chaos reigns supreme. What went wrong? It turns out, your development environment was a utopian paradise compared to the harsh realities of production. Different configurations, dependencies and environments can turn your code into a ticking time bomb. It’s like sending a penguin to the desert and expecting it to thrive — it’s just not going to happen.

Scenario 2: The “Mystery Bug” Saga

You’re on a mission to hunt down a pesky bug that’s been wreaking havoc in your production environment. You scour through logs, comb through code, and even consult the ancient scrolls of Stack Overflow. But no matter how hard you try, the bug remains elusive, like a ghost haunting your server room. The culprit? Dev/Prod disparity strikes again. What worked perfectly in your development environment is now causing chaos in production, thanks to subtle differences in configurations or dependencies. It’s like trying to solve a murder mystery where the clues keep changing every time you look away. Without parity between environments, you’re left scratching your head and cursing the heavens for your misfortune.

Scenario 3: Configuration Drift in Terraform Environments

Imagine you’ve diligently crafted your Terraform infrastructure code to provision resources for both your development and production environments. However, over time, subtle differences creep in between these environments due to manual changes or updates made directly in the console. This configuration drift can lead to inconsistencies and unexpected behavior when deploying new features or scaling up/down resources. For instance, let’s say in your development environment, you’ve set up a smaller instance type for cost efficiency, while in production, you’ve opted for larger instances to handle higher traffic loads. If these configurations aren’t reflected accurately in your Terraform code, you risk deploying changes that work perfectly in development, but fail in production, disrupting service availability.

In a nutshell, ensuring “Dev/Prod Parity” within your Terraform codebase is paramount for maintaining consistency and predictability across environments. By adhering to this 12 factor principle, you safeguard against the insidious drift that often plagues development and production setups. Imagine your development environment humming along smoothly with cost-efficient setups, only to encounter deployment nightmares when transitioning to production due to unnoticed disparities. With Dev/Prod Parity, you mitigate these risks, ensuring seamless scalability and reliable deployments. So, let’s champion this principle in our Terraform workflows, empowering teams to confidently manage infrastructure with cohesion and clarity, from development inception to production excellence.

Logs — Treat logs as event streams

They are the breadcrumbs of your infrastructure fairy tale — they lead you from the beginning to the happily ever after. But beware the breadcrumbs that lead you astray — keep your logs clear and your story straight!

Scenario 1: The Mystery of the Vanishing Logs

Picture this: you’ve just deployed a shiny new microservice into the Cloud, confident that it’ll revolutionize your company’s workflow. But as soon as it goes live, something strange happens — your logs start disappearing faster than socks in a dryer. You check your logging platform, only to find cryptic error messages like “404: Logs not found.” It’s like trying to solve a mystery with no clues and no detective. Without proper logs, debugging becomes a game of “Where’s Waldo?”, except Waldo is your crucial debugging information, and he’s nowhere to be found.

Scenario 2: The Great Flood of Logs

You’ve finally tracked down the source of that pesky bug that’s been plaguing your application for weeks. Victory is within your grasp…until you realize your logging system is spewing out more messages than a malfunctioning sprinkler system. Your logs are flooding your monitoring tools faster than you can say “log rotation.” It’s like trying to take a sip of water from a fire hose — overwhelming, messy and definitely not sustainable. Sorting through this deluge of information is like searching for a needle in a haystack, except the haystack is made of more needles than a porcupine on a sewing spree.

Scenario 3: Security Risks Due to Insecure Log Handling

Another common challenge faced by Cloud and DevOps engineers in Terraform environments is the potential security risks associated with insecure log handling practices. Inadequate protection of sensitive information, such as API keys, passwords, or access tokens, within log files can expose the infrastructure to unauthorized access and data breaches. Consider a scenario where a Terraform script responsible for provisioning Cloud resources inadvertently logs confidential credentials in plaintext format. Without proper encryption or masking mechanisms in place, these sensitive details become vulnerable to exploitation by malicious actors, compromising the integrity and confidentiality of the entire infrastructure.

In conclusion, implementing the Logs principle of the 12 Factor App within Terraform scripts is paramount for ensuring robust security measures. By adhering to this principle, we mitigate the significant risk posed by insecure log handling practices. Safeguarding sensitive information like API keys and passwords within logs is imperative to prevent unauthorized access and potential data breaches. Incorporating encryption or masking mechanisms within Terraform code enhances confidentiality and integrity, fortifying the entire infrastructure against malicious exploits. As Cloud and DevOps practitioners, embracing the Logs principle not only aligns with best practices, but also underscores our commitment to building resilient and secure Cloud environments.

Admin processes — Run admin/management tasks as one-off processes

Admin processes are the gatekeepers of your infrastructure castle — they hold the keys to the kingdom and keep the riff-raff at bay.

Scenario 1: The “Sudo Spree” Syndrome

Imagine you’re the proud owner of the keys to the kingdom, bestowed upon you by the mighty ‘sudo’ command. You wield this power like a medieval knight with a broadsword, granting you access to every nook and cranny of your Cloud kingdom. But with great power comes great temptation, and soon you find yourself on a ‘sudo spree,’ granting elevated privileges left and right faster than a squirrel on a caffeine high. Before you know it, chaos ensues. Resources are misconfigured, security holes abound, and your once pristine infrastructure resembles a medieval battlefield. It’s a cautionary tale of the dangers of unchecked admin access, where even the mightiest knights can fall prey to the allure of absolute power.

Scenario 2: The “Secrets Stash” Conundrum

In the quest to secure your kingdom, you’ve amassed a treasure trove of secrets: API keys, passwords, encryption keys, you name it. Like a dragon guarding its hoard, you stash these secrets away in the deepest, darkest corners of your infrastructure, hoping to keep them safe from prying eyes. But as any seasoned adventurer will tell you, secrets have a way of slipping through the cracks. Whether it’s a misplaced configuration file or a disgruntled employee turned rogue, your secrets are as vulnerable as a castle made of sand. And when they inevitably fall into the wrong hands, it’s not just your kingdom that’s at risk, but the entire realm of Cloud and DevOps. It’s a tale as old as time: the struggle to balance security with accessibility, where even the mightiest fortresses can crumble at the slightest misstep.

Scenario 3: Lack of Immutable Infrastructure in Terraform Workflows

Another challenge faced by Cloud and DevOps engineers is maintaining immutable infrastructure while managing Terraform workflows, which aligns with the Admin processes principle. In traditional setups, updates to infrastructure are often made in-place, directly modifying existing resources. However, this approach can introduce inconsistencies and make it challenging to rollback changes if issues arise. Suppose you’re deploying a Kubernetes cluster using Terraform, and during an update, a misconfiguration causes instability. Reverting to the previous state becomes cumbersome without proper versioning and immutable infrastructure practices.

// Example Terraform code for deploying a Kubernetes cluster
// using immutable infrastructure
// Define Kubernetes cluster resources
resource "aws_eks_cluster" "example" {
  name = "example-cluster"
  role_arn = aws_iam_role.example.arn
  version = var.kubernetes_version
  vpc_config {
  subnet_ids = var.subnet_ids
    }
}
// Define immutable update process using Terraform
resource "null_resource" "k8s_update" {
  triggers = {
  eks_cluster_version = aws_eks_cluster.example.version
  desired_version = var.desired_kubernetes_version
    }
  provisioner "local-exec" {
  command = "kubectl apply -f k8s_manifests/"
    }
  provisioner "local-exec" {
  command = "kubectl rollout restart deployment - all"
    }
}

To mitigate this, it’s crucial to adopt immutable infrastructure principles in Terraform workflows. Rather than modifying existing resources, treat infrastructure as disposable and recreate it entirely for updates.

In conclusion, the Admin processes principle of the 12 Factor App highlights the significance of treating administrative tasks as one-off processes that run against a single, immutable release. When applied to Terraform workflows, this principle underscores the importance of maintaining immutable infrastructure to ensure consistency and reliability. By adhering to immutable infrastructure practices, Cloud and DevOps engineers can mitigate the risks associated with in-place updates, enabling smoother rollbacks and better management of resources. Embracing this principle not only enhances the stability of infrastructure but also streamlines the management of Terraform code, empowering teams to confidently navigate the complexities of Cloud provisioning.

In wrapping up, adopting the 12 Factor App principles in Terraform development is more than just a trend — it’s a game-changer for Cloud and DevOps engineers. These principles offer a roadmap for building resilient, scalable and maintainable infrastructure in the Cloud era. By applying concepts like declarative formats, dependency management, and strict isolation, teams can elevate their Terraform workflows to new heights of efficiency and reliability.

Whether you’re provisioning Kubernetes clusters or spinning up serverless functions, integrating these principles ensures consistency across environments, and simplifies the management of complex infrastructure-as-code projects. From enhancing collaboration to enabling seamless deployments, the 12 Factor App principles serve as a guiding light for modern Cloud architecture.

So, as we forge ahead in this era of rapid innovation, let’s not forget the timeless wisdom embedded in these principles. Let’s embrace them, iterate upon them, and together, let’s build the resilient infrastructure of tomorrow, one Terraform module at a time.

More Read

Part 1

Part 2

Part 3

12 Cloud Commandments: Applying 12 Factor App Principles to Master Terraform — Part 3

Manish R Warang — Wed, 05 Jun 2024 10:48:02 +0000

12 Cloud Commandments: Applying 12 Factor App Principles to Master Terraform — Part 3

In the previous part of this series, we explored the initial steps to applying the 12 Factor App principles in Terraform code. We discussed:

Config : Emphasizing the separation of configuration from code, enabling dynamic and environment-specific settings. This enhances the reliability and scalability of your infrastructure provisioning process.
Backing Services : Treating backing services such as databases as attached resources. This involves securely managing credentials and leveraging tools like AWS Secrets Manager to ensure portability and maintainability.
Build, Release, Run : Strictly separating these stages to prevent configuration drift and maintain consistency across environments. This ensures what you build is exactly what you release and run, fostering a robust deployment process.

These foundational principles help in creating a scalable, maintainable, and secure infrastructure using Terraform.

Processes — Execute the app as one or more stateless processes

Choreographers of your Cloud ballet — they keep everyone in step and moving in the right direction. But watch out for the prima donna who insists on stealing the spotlight — keep your processes lean and your performance flawless!

Scenario 1: The “Zombie Apocalypse” of Processes

Imagine this: you’re managing a fleet of Cloud servers, each running multiple services to keep your application humming along. But as time goes by, you start noticing something eerie — zombie processes lurking in the shadows. These undead remnants of past deployments refuse to die gracefully, hogging precious resources and haunting your system like ghosts of past deployments. They slow down your servers, drain your budgets, and turn your Cloud infrastructure into a graveyard of wasted compute power. It’s a nightmare straight out of a horror movie, except instead of brains, these zombies crave CPU cycles.

Scenario 2: The “Whack-a-Mole” Conundrum

Ever felt like you’re playing an endless game of whack-a-mole with your Cloud infrastructure? You squash one pesky process causing trouble, only for another one to pop up somewhere else. It’s like trying to plug leaks in a sinking ship with duct tape — a never-ending cycle of firefighting and frustration. You scale up to handle increased traffic, only to find that your processes aren’t playing nice with each other, leading to bottlenecks and slowdowns. It’s enough to make even the most seasoned DevOps engineer feel like they’re chasing their own tail in a never-ending game of cat and mouse.

Scenario 3: Unmanaged Background Processes

In a typical Cloud infrastructure setup managed through Terraform, engineers often need to provision various resources like virtual machines, databases, and networking components. Sometimes, these resources require background processes for tasks such as data migration, log streaming, or periodic clean-up. However, if these background processes are not properly managed, they can lead to resource wastage, performance issues, or even unexpected downtime. Consider a scenario where a DevOps team provisions a set of EC2 instances using Terraform to host a microservices architecture. Each instance needs to run a background process for log aggregation and forwarding to a centralized monitoring system. Without proper management, these background processes might consume excessive CPU or memory, impacting the performance of the microservices. Furthermore, if one of these processes crashes or hangs, it can disrupt the entire application’s functionality.

In the realm of Terraform coding for Cloud infrastructure, the “Processes” principle from the 12 Factor App offers a crucial beacon of guidance. With its focus on executing tasks as stateless, isolated processes, it serves as a guardrail against resource inefficiencies and potential system failures. Picture this: DevOps teams spinning up EC2 instances, each necessitating background processes for vital functions like log aggregation. Without adherence to this principle, these processes could turn into resource hogs, jeopardizing the performance of our microservices. But by embracing the 12 Factor approach, we ensure that our processes remain lightweight, scalable and resilient. So, as we code our Terraform configurations, let’s keep the essence of “Processes” alive, safeguarding our infrastructure’s stability and efficiency.

Port Binding — Export services via port binding

Port binding is like musical chairs for your applications — they need a seat at the table to make beautiful music together. But don’t let them fight over the best spot — keep your ports organized and your applications harmonizing like a well-tuned orchestra!

Scenario 1: The “Port Collision” Conundrum

Imagine this scenario: you’re deploying a shiny new microservice into your Cloud environment, complete with all the bells and whistles. Everything seems to be going smoothly until you hit a snag — port collision. You see, in the wild world of Cloud infrastructure, ports are like prime real estate — everyone wants a piece of the action. But when two services try to stake their claim on the same port, chaos ensues. It’s like trying to fit two puzzle pieces into the same slot — it just doesn’t work.

As a DevOps engineer, you find yourself playing referee in this port showdown, trying to untangle the mess without disrupting the flow of traffic. You’re juggling firewall rules, network configurations, and angry service owners, all while praying that your troubleshooting skills are sharper than a katana. It’s a high-stakes game of “Portopoly,” where one wrong move could send your entire infrastructure crashing down like a house of cards.

Scenario 2: The “Ephemeral Port Panic” Predicament

Picture this: you’ve spent weeks fine-tuning your Cloud infrastructure, optimizing every last byte for peak performance. But just when you think you’ve got it all figured out, along comes the dreaded ephemeral port panic. Ephemeral ports are like the wild cards of the networking world — they come and go as they please, leaving chaos in their wake. One minute, your service is happily chugging along, and the next, it’s stuck in a port purgatory, unable to communicate with the outside world.

As a Cloud and Infrastructure Architect, you find yourself on the front lines of this ephemeral port panic, desperately trying to keep your services afloat amidst a sea of random port assignments. You’re wrangling load balancers, tweaking security groups, and praying to the demo gods for mercy, all while cursing the ephemeral nature of it all. It’s a wild ride through the port jungle, where the only law is Murphy’s Law — if something can go wrong, it will.

Scenario 3: Managing Port Allocation in Load Balancing Environments

In a load-balanced environment, ensuring proper port binding is crucial for efficient traffic distribution and high availability. Without adhering to the 12 Factor App principle of Port Binding, you might face challenges in managing port allocations across different instances of your application behind the load balancer. For instance, if you’re using Terraform to provision EC2 instances in AWS and configure them behind an Elastic Load Balancer (ELB), specifying static ports in your Terraform code can lead to port conflicts or inefficient resource utilization. Consider a scenario where you have multiple EC2 instances serving the same application behind an ELB. Without dynamic port binding mechanisms, you might configure each instance to listen on a predefined port, such as port 80. However, as your application scales and additional instances are launched, conflicts may arise as each instance competes for the same port. By following the Port Binding principle and utilizing features like dynamic port mapping in Terraform, you can ensure that each instance binds to a unique port dynamically, allowing the load balancer to efficiently distribute traffic without encountering port conflicts.

In conclusion, adopting the Port Binding principle within your Terraform code is essential for seamless scalability and efficient resource utilization in Cloud environments. By dynamically assigning ports to instances behind load balancers, you mitigate the risk of port conflicts and ensure optimal traffic distribution. Terraform’s dynamic port mapping capabilities empower you to embrace this principle effectively, enabling smoother deployments and enhancing the overall reliability of your infrastructure. Embracing Port Binding not only aligns with 12 Factor App principles, but also fosters agility and resilience within your Cloud architecture. So, let’s bind wisely, scale effortlessly, and keep our applications running smoothly in the Cloud.

Concurrency — Scale out via the process model

The juggling act of your infrastructure circus — it’s all about keeping multiple balls in the air without dropping a single one. But watch out for the clown who thinks they can juggle flaming torches — keep your concurrency safe and your audience entertained!

Scenario 1: The “Racecar Deployment” Conundrum

Imagine you’re deploying updates to your Cloud infrastructure, sprinting towards the finish line like a Formula 1 driver. Everything seems to be going smoothly, until you hit a curveball: multiple engineers deploying changes simultaneously. Suddenly, it’s less like a race and more like rush hour traffic in downtown LA. Requests collide, resources clash, and chaos ensues. You find yourself stuck in a deadlock, waiting for one deployment to finish before another can even start. It’s a racecar deployment, where everyone’s trying to be the first across the finish line, but nobody’s getting anywhere fast.

Scenario 2: The “Herding Cats” Debacle

Ever tried to coordinate with a group of DevOps engineers, each with their own agenda and timeline? It’s like herding cats on a caffeine bender. One engineer wants to deploy updates to the database, while another insists on rolling out changes to the networking infrastructure. Meanwhile, you’re stuck in the middle, trying to keep everyone moving in the same direction. But just when you think you’ve got everyone on the same page, someone decides to throw a spanner in the works by deploying their changes without warning. It’s a circus of chaos, where coordination is about as likely as finding a unicorn in your server room.

Scenario 3: Resource Collisions during Parallel Execution

Imagine a scenario where a team of DevOps engineers is managing a large-scale infrastructure on AWS using Terraform. They have multiple modules defining various resources like EC2 instances, RDS databases, and security groups. Each engineer works on different modules simultaneously to speed up development and deployment. However, without careful coordination, this concurrent development can lead to resource collisions. For example, two engineers might inadvertently attempt to create resources with the same name or in the same subnet, causing conflicts and potentially breaking the infrastructure. To address this, engineers must implement strategies to ensure resource isolation and prevent collisions. One approach is to use Terraform workspaces or state locking mechanisms provided by Terraform backends like Amazon S3 or HashiCorp Consul. By utilizing these features, engineers can ensure that only one process can modify the state at a time, preventing conflicts and maintaining the integrity of the infrastructure.

In conclusion, while the allure of concurrent development in Cloud infrastructure management is undeniable for accelerating deployment, it introduces a critical challenge: concurrency. Without proper handling, simultaneous work on Terraform modules can lead to resource collisions, jeopardizing the stability of the infrastructure. Fortunately, by embracing the 12 Factor App principle of Concurrency, DevOps teams can implement strategies like Terraform workspaces and state locking mechanisms to ensure resource isolation and prevent conflicts. These measures not only maintain the integrity of the infrastructure but also foster smoother collaboration among engineers. So, remember, when it comes to managing Cloud infrastructure with Terraform, prioritizing concurrency management isn’t just a best practice — it’s a necessity for sustained efficiency and reliability.

More Read

Part 1

Part 2

Part 4

12 Cloud Commandments: Applying 12 Factor App Principles to Master Terraform — Part 2

Manish R Warang — Tue, 28 May 2024 07:20:31 +0000

12 Cloud Commandments: Applying 12 Factor App Principles to Master Terraform — Part 2

In Part 1, we introduced the integration of the 12 Factor App principles with Terraform to optimize Cloud infrastructure management. We began with an overview of these principles, highlighting their relevance in building scalable and maintainable applications.

The Codebase principle emphasized maintaining a single codebase with multiple deploys, ensuring a clean and organized environment. We also discussed the Dependencies principle, focusing on the need to explicitly declare and isolate dependencies to prevent hidden or unmanaged issues.

These foundational concepts set the stage for effective Infrastructure as Code practices using Terraform.

Config — Store configuration in the environment

Configurations are the seasoning of your infrastructure — they add flavor and personality, but too much can leave a bad taste in your mouth. Keep it simple, sprinkle just enough to enhance the flavor, and avoid drowning your dish in a sea of spices. Your infrastructure will thank you for it!

Scenario 1: The “Configuration Chaos Carnival”

Ever felt like you’re juggling a dozen flaming torches while riding a unicycle on a tightrope? Welcome to the Configuration Chaos Carnival! You have configurations spread across multiple files, environments, and even secret management systems. Keeping track of which setting applies where is like herding cats during a fireworks display — chaotic, unpredictable and downright dangerous. One wrong move could set off a chain reaction of misconfigurations, turning your infrastructure into a virtual circus of errors. It’s enough to make even the most seasoned Cloud or DevOps engineer question their sanity.

Scenario 2: The “Secrets Slip-Up”

Picture this: you’re deploying a critical application to production, and everything is going smoothly. But as you reach the final stages, you realize you forgot to properly manage your secrets. Your database passwords are hardcoded in plain text, API keys are floating around in Slack channels, and SSH keys are stored in a folder cleverly named “Not_Secrets.” It’s a security nightmare waiting to happen, like leaving the keys to your Ferrari in the ignition with a sign saying “Free Joyrides.”

Scenario 3: Environment-specific Configuration

Another challenge arises when managing environment-specific configuration in Terraform. In a typical application deployment pipeline, you may have multiple environments such as development, staging and production. Each environment requires different configuration settings, such as database endpoints, API URLs or feature flags. Hardcoding these settings directly into Terraform code leads to maintenance overhead and potential errors when promoting code across environments.

resource "aws_instance" "example" {
  ami = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
  subnet_id = "subnet-029b2e75"
  security_groups = ["${var.security_group_name}"]
  tags = {
  Name = "ExampleInstance"
  }
}

In this example, the subnet_id and security_group_name are hardcoded. While this may work for a specific environment, it becomes cumbersome to manage when deploying to different environments with unique network configurations.

In conclusion, adhering to the “Config” principle of the 12 Factor App is paramount, especially in the context of Terraform code. By separating configuration from code, we mitigate the headaches of managing environment-specific settings. Leveraging tools like Terraform’s input variables, or even external configuration management systems, enables seamless configuration across development, staging and production environments. This approach not only streamlines maintenance, but also enhances the reliability and scalability of our infrastructure provisioning process. Embracing the “Config” principle empowers DevOps teams to efficiently manage dynamic infrastructure requirements while fostering a culture of agility and resilience.

Backing Services — Treat backing services as attached resources

They are like your trusty sous chefs — they handle the grunt work so you can focus on the main course. Whether it’s a database slicing and dicing your data or a CDN spreading your content far and wide, choose your sidekicks wisely. After all, no one wants a sous chef who burns the soufflé!

Scenario 1: The Vanishing Database Conundrum

Imagine this scenario: you’re in the final stages of deploying a new application to the Cloud. Everything seems to be going smoothly` until it’s time to connect to the database. You enter the credentials provided by your colleague, only to be met with a dreaded error message: “Connection refused.” After several frantic Slack messages and a few cups of coffee, you discover that the database instance has mysteriously vanished into the digital ether. Turns out, your colleague forgot to renew the subscription, and now you’re left scrambling to spin up a new instance, and migrate the data before the stakeholders start breathing down your neck. It’s like trying to catch smoke with a butterfly net — frustrating, futile, and just a tad ridiculous.

Scenario 2: The Shadowy Service Blackout

Picture this: you’re knee-deep in troubleshooting a production issue that’s causing your application to crash more frequently than a Windows 95 computer. After hours of digging through logs and scratching your head, you finally uncover the culprit — a third-party service that your application relies on for critical functionality. But here’s the kicker: the service provider has suddenly gone dark, with no updates on their status page, and no response to your frantic support tickets. Now you’re stuck in a digital purgatory, waiting for the service to resurface like a submarine in distress. Meanwhile, your users are flooding your inbox with complaints, and your boss is giving you the stink eye from across the office. It’s a lesson in the importance of vetting and monitoring backing services, lest you find yourself adrift in a sea of uncertainty.

Scenario 3: Managing Database Credentials in Terraform Code

DevOps engineers face challenges in securely managing database credentials in Cloud infrastructure provisioning. To follow the 12 Factor App’s “Backing Service” principle, backing services like databases should be treated as attached resources. However, hardcoding sensitive information into Terraform configurations violates security best practices and makes the infrastructure less portable and maintainable. To address this, Terraform can be used to retrieve database credentials from secure secret management services like AWS Secrets Manager or HashiCorp Vault, separating infrastructure provisioning concerns from sensitive data management, adhering to the principle of treating backing services as attached resources.

`
data "external" "database_credentials" {
program = ["bash", "${path.module}/scripts/get_database_credentials.sh"]
}

resource "aws_db_instance" "example" {

Other configuration options…

username = data.external.database_credentials.result.username
password = data.external.database_credentials.result.password

Database instance configuration…

}
`

In this example, get_database_credentials.sh script retrieves database credentials from a secure source and outputs them in a format that Terraform can consume. By separating credential management from infrastructure code, you adhere to the principles of security, scalability, and portability advocated by the 12 Factor App.

In wrapping up, the “Backing Services” principle from the 12 Factor App provides a crucial framework for managing Infrastructure resources, e.g. databases securely in Cloud infrastructure provisioning. DevOps engineers often grapple with the challenge of safeguarding sensitive credentials, while ensuring infrastructure portability and maintainability. Terraform emerges as a powerful ally in this regard, enabling the separation of concerns by retrieving credentials from secure secret management services like AWS Secrets Manager or HashiCorp Vault. By adhering to this principle, we not only enhance security, but also streamline the management of backing services, fostering a robust and agile Cloud infrastructure ecosystem. Let’s embrace Terraform’s potential to uphold these principles and propel our Cloud and DevOps endeavors to new heights.

Build, Release, Run — Strictly separate build and run stages

Ah, the three musketeers of deployment — build, release and run. Like a well-choreographed dance routine, they work in harmony to bring your creation to life. But beware the rogue dancer who steps on everyone’s toes — keep your deployments smooth and your audiences applauding!

Scenario 1: The “Release Roulette”

Imagine this scenario: you’re preparing to deploy a new feature to production. You’ve meticulously built and tested the code on your local machine, but as soon as it hits the production environment, chaos ensues. The application crashes, users start complaining, and you’re left scrambling to figure out what went wrong. Turns out, the code that worked perfectly in your development environment doesn’t play nice with the dependencies and configurations in the production environment. It’s like trying to fit a square peg into a round hole, except the peg is your code and the hole is production. This release roulette not only disrupts the user experience, but also leaves you questioning your life choices as a DevOps engineer.

Scenario 2: The “Version Vortex”

Picture this: you’re knee-deep in managing multiple versions of your application across different environments. You have one version running in development, another in staging, and a different one in production. Keeping track of which version is where feels like playing a game of “Whac-A-Mole” with your sanity. You make a change to fix a bug in the staging environment, only to realize it’s the wrong version, and now you’ve introduced a new bug into production. It’s a never-ending version vortex that sucks you in deeper with each deployment, leaving you feeling more lost than a tourist without a map. Trying to maintain consistency and control across environments becomes a constant battle, with version numbers swirling around you like a tornado of confusion.

Scenario 3: Configuration Drift in Production Environment

Cloud and DevOps engineers face the challenge of maintaining consistency between different environments, particularly in dynamic Cloud infrastructure. The “Build, Release, Run” principle is crucial in ensuring environment parity by separating the build phase, where infrastructure configurations are defined in Terraform code, from the release and run phases, where these configurations are applied to different environments. This helps mitigate the risk of configuration drift.

A continuous delivery approach with Terraform pipelines enables automated deployments from a centralized source of truth, such as a version-controlled repository. Changes to the infrastructure are tested in lower environments before being promoted to production, reducing the likelihood of unexpected configuration drift. Tools like Terraform Enterprise offer features like state locking and version control, providing visibility and control over changes made to the production environment, thus maintaining consistency and reliability.

In conclusion, adopting the “Build, Release, Run” principle within Terraform coding not only addresses the challenge of maintaining consistency across various Cloud environments, but also fortifies the reliability of your infrastructure. By segregating the build phase from release and run phases, you effectively combat configuration drift, ensuring that what you build is exactly what you release and run. Embracing continuous delivery methodologies through Terraform pipelines automates deployments, fostering a culture of agility and reliability. With tools like Terraform Enterprise providing robust features such as state locking and version control, you gain unparalleled visibility and control over your infrastructure changes. In essence, integrating 12 Factor App principles into Terraform workflows elevates not just efficiency, but the integrity of your Cloud infrastructure.

More Reads

Part 1

Part 3

Part 4

12 Cloud Commandments: Applying 12 Factor App Principles to Master Terraform — Part 1

Manish R Warang — Tue, 21 May 2024 06:32:51 +0000

12 Cloud Commandments: Applying 12 Factor App Principles to Master Terraform — Part 1

Remember the days when deploying a new app meant wrestling with servers, deciphering cryptic configuration files, and hoping you didn’t trigger a rogue hamster on the wheel powering your infrastructure? Those days, thankfully, are fading faster than a Snapchat filter. Modern Cloud architecture and DevOps practices have evolved into a sleek, efficient dance between code and infrastructure. We’re talking automation, standardization, and a whole lot less server-induced sweat.

But amidst this technological tango, one set of principles still shines brightly: the 12 Factor App. These timeless guidelines, born in the early days of the Cloud, remain as relevant as ever. They’re like the secret sauce that keeps your applications scalable, reliable and deployable with the click of a button.

So, buckle up, because, in this blog post, we’re about to do something pretty cool: we’re going to marry the power of Terraform, an open-source Infrastructure as Code (IaC) tool, with the wisdom of the 12 Factor App principles. Think of it as a match made in DevOps heaven. We’ll show you how each principle can be applied within the context of Terraform, creating an infrastructure that’s as nimble and adaptable as your code itself.

12 Factor App Principles

The twelve-factor app is a methodology for building software-as-a-service apps that are designed to be:

Scalable
Maintainable
Portable

The principles guide developers in designing Cloud-native applications, focusing on codebases, dependencies, configuration and processes, ensuring productivity, dynamic scaling and resilience, and promoting a consistent approach to modern Cloud platforms.

Some key aspects of the twelve-factor methodology include:

Strict separation of config from code
Ability to scale out via the process model
Loose coupling between app components
Designing stateless processes
Ease of deploying/rolling back versions
Minimizing divergence between development, staging, production
Easy portability between execution environments

We will now examine the amalgamation of two potent techniques in this blog post: Terraform, an open-source Infrastructure as Code (IaC) tool, and the “12 Factor App” principles. This union provides a significant synergy that enables us to create robust repeatable infrastructure at scale.

So, let’s get started, shall we?

Codebase — One Codebase Tracked in Revision Control, Many Deploys

Think of your codebase as the ultimate recipe for your Cloud infrastructure. Just like a well-loved cookbook, it should be organized, easy-to-follow, and free of spaghetti code. When everyone’s on the same page and singing from the same hymn sheet, you’ll avoid the chaos of mismatched ingredients and cooking disasters. Keep it tidy, folks — no one likes a messy kitchen!

Scenario 1: The “Spaghetti Stack” Dilemma

Picture this: you inherit a project from a colleague who left the company to pursue their dream. As you dive into the codebase, you realize it’s more tangled than a plate of spaghetti at an Italian wedding. Modules, functions, and configurations are all mixed-up, making it impossible to decipher what does what. This spaghetti stack not only confuses you, but also increases the risk of introducing bugs or making unintended changes. It’s like trying to find a needle in a haystack, except that the haystack is made of needles.

Scenario 2: The “Version Control Vortex”

Ever experienced the frustration of trying to track changes in a project where everyone seems to have their own version of reality? It’s like playing a game of “Chinese Whispers” but with code. One engineer adds a new feature in their local environment, another tweaks a configuration file on the server directly, and chaos ensues. Before you know it, you’re knee-deep in merge conflicts, lost commits and version control mayhem. It’s a nightmare for collaboration, and leads to more headaches than a hangover on a Monday morning.

Scenario 3: Terraform Configuration Monolith

A new DevOps team managing a sprawling AWS infrastructure encounters a monolithic configuration file in the Terraform codebase. This file consists of every aspect of the infrastructure, from networking to compute instances, making it intimidating and introducing risks. A simple typo could lead to critical services downtime, and understanding the impact of changes becomes a difficult task. For example, updating security group rules for a specific service can be a “Where’s Waldo” game, increasing the likelihood of human error and downtime.

The 12-Factor App principle of “Codebase” comes to the rescue. Treat your Terraform code like the precious Infrastructure as Code (IaC) it is. Store it in a central repository like Git, the knight in shining armor of version control. This allows for easy collaboration, disaster recovery, and a sigh of relief knowing you can revert to a working version, if needed.

In conclusion, embracing the “Codebase” principle of the 12 Factor App not only promotes clarity and manageability, but also mitigates risks inherent in sprawling infrastructure configurations. By modularizing Terraform code into discrete components, teams can effectively tackle complexity, reducing the likelihood of human error and downtime. A monolithic configuration file transforms into a navigable landscape, where updates are precise, and impact assessments are straightforward. With this approach, managing AWS infrastructure becomes less of a daunting task and more of a streamlined operation, empowering DevOps teams to wield Terraform with confidence and efficiency.

Dependencies — Explicitly declare and isolate dependencies

They are like the ingredients in your favorite dish — you need them to make the magic happen, but too many can spoil the broth. Keep a close eye on what you’re adding to the pot, and make sure each ingredient pulls its weight. After all, no one wants a soufflé that collapses under the weight of too many eggs!

Scenario 1: The “Dependency Domino Effect”

You’re happily coding away, when suddenly a wild dependency appears! You install it without a second thought, only to realize it brings along its own entourage of dependencies. Before you know it, your project has more dependencies than a family reunion. Each update becomes a game of Russian roulette, as you pray that one tiny change doesn’t send the entire stack tumbling down like a house of cards. It’s dependency management, but with all the stress of a high-stakes poker game.

Scenario 2: The “Versioning Volcano”

Picture this: you’re tasked with updating a library in your project to fix a critical security vulnerability. Easy, right? Wrong. Turns out, that library is a critical piece of the puzzle, used in multiple modules across your application. You update it in one place, only to realize that it breaks compatibility with another module that’s still using an older version. Now you’re stuck in a versioning volcano, trying to balance stability with security while dodging eruptions left and right. It’s a precarious game of “Jenga,” where one wrong move could bring the whole tower crashing down.

Scenario 3: Uncontrolled Module Dependencies

In the world of Cloud and DevOps, managing dependencies in Terraform modules is crucial for maintaining a scalable and efficient Infrastructure as Code (IaC) setup. Consider a scenario where you have a main Terraform module that provisions AWS resources such as EC2 instances and RDS databases, and it relies on a separate module for managing security groups. Now, imagine that the security group module is updated frequently with new features or fixes. Without careful versioning and dependency management, these updates could unintentionally break the main module, leading to deployment failures or security vulnerabilities.

To illustrate, let’s say you have a Terraform configuration like this:

module "main" {
source = "./main_module"
// other configurations
security_group_id = module.security_group.id
}
module "security_group" {
source = "./security_group_module"
// other configurations
}

In this example, the main module depends on the security group module. However, if the security group module undergoes significant changes without proper versioning or testing, it could introduce unexpected behavior or errors in the main module, disrupting the infrastructure deployment process.

In the ever-evolving landscape of Cloud and DevOps, mastering dependency management in Terraform is non-negotiable. Picture this: your main Terraform module orchestrating vital AWS resources is only as robust as its supporting modules, like those handling security groups. Yet, without diligent versioning and dependency control, each update to these ancillary modules becomes a high-stakes gamble, risking deployment hiccups or worse, security breaches. Embracing the 12 Factor App principle of Dependencies isn’t just about ticking boxes; it’s about safeguarding your infrastructure’s integrity and agility. So, let’s commit to meticulous dependency management, ensuring our Terraform setups scale reliably in the face of constant change.

More Read

Build AWS AMI with HashiCorp Packer using GitHub Actions

Manish R Warang — Tue, 10 Aug 2021 07:02:53 +0000

Security best practices recommend the latest base images (e.g. AWS AMI ) for spinning VM on the cloud. Up to date software patches reduce the risk of any security breach. This emphasizes a strong need to set up an Image Factory to automate the image creation process with the latest software versions.

Also, every organization has a standard list of software to be installed for a given image. Only the essential software must be part of the list to reduce the blast surface from a security standpoint. This list of software can either be installed once the VM is created or when the VM is in the process of creation. However, both these techniques are time-consuming and hence not recommended.

It would be preferred to bake this software list in the base image itself. HashiCorp Packer is an open-source tool that specializes in building automated machine images for multiple platforms from a single source configuration.

HashiCorp Packer Installation

Refer to HashiCorp documentation for Packer installation based on your hardware OS.

GitHub Actions

For this demo, we will use GitHub Actions to create CI/CD pipeline to automate this workflow and eventually push the baked image (AMI) in AWS. It is a platform to automate tasks within the software development lifecycle. It's an event-driven framework, which means we can carry series of commands for a given event or can be scheduled for one-off or repetitive tasks. (e.g. Execute a Test Suite on Pull Request creation, Adding labels to issues, Lint checks, etc.)

Actions are defined in YAML files, which allows pipeline workflow to be triggered using any GitHub events like on creation of Pull Requests, on code commits, and much more.

Prerequisites

AWS User with Programmatic access
- AWS Access Key ID
- AWS Secret Access Key
AWS IAM Privileges to create EC2 Instance (create, modify and delete EC2 instances). Refer documentation for the full list of IAM permissions required to run the amazon-ebs builder.

In this post, we will bake Open JDK (Java 8) in our Ubuntu base Image and push it into AWS. Packer configurations can be written in HCL (.pkr.hcl file extension) and JSON (.pkr.json) formats. We will use the HCL language for this demo.

Reference GitHub repository - pkr-aws-ubuntu-java

Code Time

Let us start writing Packer configuration. (I am using a Linux machine for this demo)

Packer Configuration

Create Project folder pkr-aws-ubuntu-java

mkdir pkr-aws-ubuntu-java && cd $_

Create a file named aws-demo.pkr.hcl

touch aws-demo.pkr.hcl

Open your favorite IDE (e.g. VSCode). Copy the below code in aws-demo.pkr.hcl file.

packer {
  required_plugins {
    amazon = {
      version = ">= 0.0.2"
      source = "github.com/hashicorp/amazon"
    }
  }
}

The packer {} block contains Packer settings, including a required Packer version. The required_plugins block in the Packer block, specifies the plugin required by the template to build your image. The plugin block contains a version and source attribute.

Source block

The source block configures a specific builder plugin, which is then invoked by the build block. Source blocks use builders and communicators to define virtualization type, image launch type, etc.

Copy the following code to aws-demo.pkr.hcl file.

variable "ami_prefix" {
  type = string
  default = "packer-aws-ubuntu-java"
}

locals {
  timestamp = regex_replace(timestamp(), "[- TZ:]", "")
}

source "amazon-ebs" "ubuntu_java" {
  ami_name = "${var.ami_prefix}-${local.timestamp}"
  instance_type = "t2.micro"
  region = "us-east-1"
  source_ami_filter {
    filters = {
      name = "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*"
      root-device-type = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners = ["099720109477"]
  }
  ssh_username = "ubuntu"
}

Variable ami_prefix is used to define the AMI image. Local variable timestamp helps ensure uniqueness to AMI name.

The amazon-ebs builder launches the source AMI, runs provisioners within this instance, then repackages it into an EBS-backed AMI. This builder configuration launches a t2.micro AMI in the us-east-1 region using an ubuntu:xenial AMI as the base image.

It creates the AMI named packer-aws-ubuntu-java+timestamp. AMI names must be unique else it will throw an error.

It also uses the SSH communicator - by specifying the ssh_username attribute. Packer is then able to SSH into EC2 instance using a temporary keypair and security group to provision your instances.

Build Block

The build block defines what Packer should do with the EC2 instance after it launches.

build {
  name = "packer-ubuntu"
  sources = [
    "source.amazon-ebs.ubuntu_java"
  ]

  provisioner "shell" {

    inline = [
      "echo Install Open JDK 8 - START",
      "sleep 10",
      "sudo apt-get update",
      "sudo apt-get install -y openjdk-8-jdk",
      "echo Install Open JDK 8 - SUCCESS",
    ]
  }
}

provisioner block helps automate modifications to your base image. It leverages shell scripts, file uploads, and integrations with modern configuration management tools such as Ansible, Chef, etc.

The above provisioner defines a shell provisioner and installs Open JDK 8 in the base image.

The final file aws-demo.pkr.hcl should look as below.

packer {
  required_plugins {
    amazon = {
      version = ">= 0.0.2"
      source = "github.com/hashicorp/amazon"
    }
  }
}

variable "ami_prefix" {
  type = string
  default = "packer-aws-ubuntu-java"
}

locals {
  timestamp = regex_replace(timestamp(), "[- TZ:]", "")
}

source "amazon-ebs" "ubuntu_java" {
  ami_name = "${var.ami_prefix}-${local.timestamp}"
  instance_type = "t2.micro"
  region = "us-east-1"
  source_ami_filter {
    filters = {
      name = "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*"
      root-device-type = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners = ["099720109477"]
  }
  ssh_username = "ubuntu"
}

build {
  name = "packer-ubuntu"
  sources = [
    "source.amazon-ebs.ubuntu_java"
  ]

  provisioner "shell" {

    inline = [
      "echo Install Open JDK 8 - START",
      "sleep 10",
      "sudo apt-get update",
      "sudo apt-get install -y openjdk-8-jdk",
      "echo Install Open JDK 8 - SUCCESS",
    ]
  }
}

GitHub Actions

Create a new file in the .github/workflows directory named github-actions-packer.yml

We will schedule this workflow to run in the wee hours - let's say 04:00 Hrs in the morning.

name - The name of your workflow. GitHub displays the names of your workflows on your repository's actions page - "AWS AMI using Packer Config"

name: AWS AMI using Packer Config

on - (Required) The name of the GitHub event that triggers the workflow. We have configured to trigger the workflow on schedule.

on:
  schedule:
    # * is a special character in YAML so you have to quote this string
    - cron: '0 4 * * *'

jobs - A workflow run is made up of one or more jobs. These jobs can run in parallel or sequentially. Each job executes in a runner environment specified by runs-on.

job name - The name of the job displayed on GitHub.

runs-on - (Required) Determines the type of machine to run the job on. The machine can be either a GitHub-hosted runner or a self-hosted runner. Available GitHub-hosted runner types are: windows-latest / windows-2019 / windows-2016 / ubuntu-latest / ubuntu-20.04 etc.

jobs:
  packer:
    runs-on: ubuntu-latest
    name: packer

steps - Sequence of tasks called steps within a Job. They can execute commands, set up tasks, or run actions in your repository, a public repository, or action published in a Docker registry.

The first step is to check out the source code in the runner environment.

Checkout V2- This action checks out your repository under $GITHUB_WORKSPACE, so your workflow can access it.

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

To ensure access to the AWS Cloud environment we need to configure AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the runner environment. The values for these variables will be configured as GitHub Secrets in the below section.

Configure AWS Credentials - This action configures AWS credential and region environment variables for use in other GitHub Actions.

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          # aws-session-token: ${{ secrets.AWS_SESSION_TOKEN }} 
          # if you have/need it
          aws-region: us-east-1

Init initializes the Packer configuration used in the GitHub action workflow.

      # Initialize Packer templates
      - name: Initialize Packer Template
        uses: hashicorp/packer-github-actions@master
        with:
          command: init

Validate checks whether the configuration has been properly written. It will throw an error otherwise.

      # validate templates
      - name: Validate Template
        uses: hashicorp/packer-github-actions@master
        with:
          command: validate
          arguments: -syntax-only
          target: aws-demo.pkr.hcl

Build executes the Packer configuration.

      # build artifact
      - name: Build Artifact
        uses: hashicorp/packer-github-actions@master
        with:
          command: build
          arguments: "-color=false -on-error=abort"
          target: aws-demo.pkr.hcl
        env:
          PACKER_LOG: 1

The complete file github-actions-packer.yml will look as below.

name: AWS AMI using Packer Config

on:
  schedule:
    # * is a special character in YAML so you have to quote this string
    - cron: '0 4 * * *'

jobs:
  packer:
    runs-on: ubuntu-latest
    name: packer

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          # aws-session-token: ${{ secrets.AWS_SESSION_TOKEN }} 
          # if you have/need it
          aws-region: us-east-1

      # Initialize Packer templates
      - name: Initialize Packer Template
        uses: hashicorp/packer-github-actions@master
        with:
          command: init

      # validate templates
      - name: Validate Template
        uses: hashicorp/packer-github-actions@master
        with:
          command: validate
          arguments: -syntax-only
          target: aws-demo.pkr.hcl

      # build artifact
      - name: Build Artifact
        uses: hashicorp/packer-github-actions@master
        with:
          command: build
          arguments: "-color=false -on-error=abort"
          target: aws-demo.pkr.hcl
        env:
          PACKER_LOG: 1

The source code is ready and can be pushed to the GitHub repository. As configured, the workflow will be triggered at 04:00 hrs in the morning.

Terraform -Automate CI/CD Workflows via GitHub Actions

Manish R Warang — Sun, 18 Jul 2021 14:45:39 +0000

This article will set up a CI/CD pipeline for our Terraform source code ( refer post ) to spin AWS EC2 instance. The aim is to automate our development workflow by building the DevOps pipeline using GitHub Actions.

GitHub Actions

Before we proceed further let's understand GitHub Actions. It is a platform to automate tasks within the software development lifecycle. It's an event-driven framework, which means we can carry series of commands for a given event or can be scheduled for one-off or repetitive tasks. (e.g. Execute a Test Suite on Pull Request creation, Adding labels to issues, Lint checks, etc.)

It is fully integrated into GitHub. It also gives added advantage to store the source code and the CI/CD pipeline execution on the same platform. CI/CD pipeline is one of the automation workflow offerings to streamline the overall software development and delivery process.

We can apply GitHub Actions, essentially to any stage of GitHub flow.

Configure CI/CD
Execute a specific automated task when an issue is opened
Generate automate reminders for Pull Requests based on owners or reviewers

Workflow Strategy

Github allows us to create workflows in the following ways:

Create a YAML config (file name - *.yml or *.yaml) within the GitHub repository.
Create the workflow via Actions Tab on Github Repository's Web Interface.

Clicking on Set up this workflow will pre-fill the required Terraform workflow.

For this article, we will focus on the first approach.

Prerequisites

Fork the Github Repository - tf-aws-ec2
AWS User with Programmatic access
- AWS Access Key ID
- AWS Secret Access Key
AWS IAM Privileges to create EC2 Instance

Recommended to create a feature branch and checkout this branch

git checkout -b github-actions-demo

Create a new file in the .github/workflows directory named github-actions-demo.yml

Now, let's start writing the configuration in the yaml file.

name - The name of your workflow. GitHub displays the names of your workflows on your repository's actions page - "Terraform Build Demo"

name: 'Terraform Build Demo'

on - (Required) The name of the GitHub event that triggers the workflow. We have configured to trigger the workflow on Pull Request and Push events to the main branch.

Pull Request event- Triggered when the Pull request will be raised for the new feature branch
Push event - Triggered when the Pull Request is merged into the main branch.

on:
  push:
    branches:
      - "main"
  pull_request:
    branches:
      - "main"

jobs - A workflow run is made up of one or more jobs. These jobs can run in parallel or sequentially. Each job executes in a runner environment specified by runs-on.

job name - The name of the job displayed on GitHub.

runs-on - (Required) Determines the type of machine to run the job on. The machine can be either a GitHub-hosted runner or a self-hosted runner. Available GitHub-hosted runner types are: windows-latest / windows-2019 / windows-2016 / ubuntu-latest / ubuntu-20.04 etc.

environment - The environment that the job references. All environment protection rules must pass before a job referencing the environment is sent to a runner.

jobs:
  terraform:
    name: 'TF GitHub Actions Demo'
    runs-on: ubuntu-latest
    environment: production

defaults.run - Helps define default shell and working-directory options for all run steps in a workflow.

    defaults:
      run:
        shell: bash

steps - Sequence of tasks called steps within a Job. They can execute commands, set up tasks, or run actions in your repository, a public repository, or action published in a Docker registry.

The first step is to check out the source code in the runner environment.

Checkout V2- This action checks out your repository under $GITHUB_WORKSPACE, so your workflow can access it.

    - name: Checkout
      uses: actions/checkout@v2

The second step is to set up Terraform CLI in the runner environment.

setup-terraform - is a JavaScript action that sets up Terraform CLI

    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v1
      with:
        terraform_version: 1.0.0

Configure AWS Credentials - This action configures AWS credential and region environment variables for use in other GitHub Actions.

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        # aws-session-token: ${{ secrets.AWS_SESSION_TOKEN }} 
        # if you have/need it
        aws-region: us-east-1

The runner environment is now configured. We can now configure Terraform commands.

run - Runs command-line programs using the operating system's shell.

Terraform Init initializes the configuration used in the GitHub action workflow.

- name: Terraform Init
  id: init
  run: terraform init

Terraform Format checks whether the configuration has been properly formatted. It will throw an error if the configuration isn't properly formatted.

    - name: Terraform Format
      id: fmt
      run: terraform fmt -check
      env:
        TF_ACTION_WORKING_DIR: .
      continue-on-error: true

Terraform Validate validates the configuration used in the GitHub action workflow.

- name: Terraform Validate
  id: validate
  run: terraform validate -no-color

Terraform Plan generates a Terraform plan.

This step only runs on pull requests. The PR generates a plan. When the PR is merged, that plan will be applied.
This step will continue even when it errors. This allows the next step to display the plan error message even if this step fails

Terraform Plan Status returns whether a plan was successfully generated or not.

- name: Terraform Plan Status
  if: steps.plan.outcome == 'failure'
  run: exit 1

Terraform Apply applies the configuration. This step will only run when a commit is pushed to main

- name: Terraform Apply
  if: github.ref == 'refs/heads/main' && github.event_name == 'push'
  run: terraform apply -auto-approve

The complete file github-actions-demo.yml will look as below.

name: 'Terraform Build Demo'

on:
  push:
    branches:
      - "main"
  pull_request:
    branches:
      - "main"

jobs:
  terraform:
    name: 'TF GitHub Actions Demo'
    runs-on: ubuntu-latest
    environment: production

    defaults:
      run:
        shell: bash

    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v1
      with:
        terraform_version: 1.0.0

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        # aws-session-token: ${{ secrets.AWS_SESSION_TOKEN }} 
        # if you have/need it
        aws-region: us-east-1

    - name: Terraform Init
      id: init
      run: terraform init

    - name: Terraform Format
      id: fmt
      run: terraform fmt -check
      env:
        TF_ACTION_WORKING_DIR: .
      continue-on-error: true

    - name: Terraform Validate
      id: validate
      run: terraform validate -no-color

    - name: Terraform Plan
      id: plan
      if: github.event_name == 'pull_request'
      run: terraform plan -no-color
      continue-on-error: true

    - name: Terraform Plan Status
      if: steps.plan.outcome == 'failure'
      run: exit 1

    - name: Terraform Apply
      if: github.ref == 'refs/heads/main' && github.event_name == 'push'
      run: terraform apply -auto-approve

We need to also configure AWS credentials such that they are accessible to the GitHub Actions YAML script. The best way is to configure them as GitHub Secrets against the repository.

Navigate to your GitHub repository on Web Console --> Settings --> Secrets (Left Nav Bar) --> Click New Repository Secret

Configure values for the following variables as GitHub Secrets:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
PERSONAL_ACCESS_TOKEN

For Personal Access Token (PAT) generation, refer to the following GitHub docs.

Raise Pull Request for the new branch via Web Console. Refer following GitHub docs for more information.

Once the PR is raised, the GitHub Actions Job is triggered for the Pull Request event.

On Clicking the Details link, we can see all the executed Steps and their corresponding logs.

Observations:

Terraform Plan run was successful, hence the Terraform Plan Status run execution was skipped due to the failure filter condition.
The Terraform Apply run execution was also skipped - configured to be executed on the PUSH event.

Merge Pull Requests (PUSH event)

On merging the Pull request into the main branch. The configured GitHub actions workflow will be triggered again for the PUSH event.

Click on the second workflow run of GitHub Actions Demo.

Click on TF GitHub Actions Demo.

Observations:

Terraform Plan run was skipped as it will be triggered only on PULL Request.
The Terraform Apply run was successfully executed - configured to be executed on the PUSH event. As a result, an AWS EC2 instance was created.

The existing setup is now capable of handling any changes (new/updates). The Terraform scripts will be automatically deployed to AWS Cloud once source code is merged into the main branch as demonstrated above.

Destroy resources

Remember to destroy the resources (i.e AWS EC2 instance) you created for this tutorial to avoid any costs.

Refer to the GitHub Repo for the source code demonstrated in the above post.

Terraform - Code Structure

Manish R Warang — Mon, 12 Jul 2021 16:22:16 +0000

The previous Terraform blog gave us a basic introduction to Terraform. We also discussed writing a simple Terraform code to create an AWS EC2 instance with minimal code.

Maintaining all codebases in a single main.tf file is good for beginners. However, this approach will lead to maintainability issues as our underlying infrastructure grows. Moreover, in practical situations, you might also need to deal with multiple environments (e.g. DTAP - dev/test/acceptance/production).

Other deciding factors for code modularization are as follows-

Project Complexity
- Count of Terraform Providers involved
- Count of Infra resources to be maintained by Terraform
Cadence of Infrastructure changes - daily / weekly / monthly
Deployment Strategy - CI/CD Pipeline, GitOps, etc.

It is recommended to logically split the source code as follows:

provider.tf - contains provider configuration in root module
main.tf - call modules, locals, and data sources to create all resources
variables.tf - contains variable declarations used in main.tf
outputs.tf - contains outputs from the resources created in main.tf
terraform.tfvars - contains variable definitions to provide default variable values. Terraform will automatically load variables from those files.

Let us refer to the initial Terraform code and understand how we can logically break it down further.

As shown in the above image,

Red block will be part of provider.tf
Green block will be part of main.tf

Using Input Variable Values

As we treat Terraform as IaC, we should ensure there are no hardcodings. These must be configured in variables.tf. The following block must be defined for every variable.

variable "aws_region" {
    type = string
    description = "AWS Region"
# default value is optional.
    default = "us-east-1"
}

Variables on the Command Line

A default value can be configured (as shown above) for a variable. However, we can override this value while executing the code at runtime.

terraform apply -var aws_region="eu-west-1"

Environment Variables

Also, Terraform can search the environment of its own process for environment variables named TF_VAR_ followed by the name of a declared variable. This can be useful when running Terraform in automation.

export TF_VAR_aws_region = eu-west-1

Variable Definitions (.tfvars) Files

For bulk values, it is of convenience to specify their values in a variable definitions file (with a filename ending in either .tfvars or .tfvars.json)

Terraform also automatically loads a number of variable definitions files if they are present:

Files named exactly terraform.tfvars or terraform.tfvars.json.
Any files with names ending in .auto.tfvars or .auto.tfvars.json.

This variable can then be used to replace the hardcodings.

region = var.aws_region

Our code post formatting will look as follows:

provider.tf

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "~>3.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
  default_tags {
    tags = {
      "Environment" = "dev"
      "Owner" = "g33kzone"
    }
  }
}

variables.tf

variable "aws_region" {
    type = string
    description = "AWS Region"
}

variable "instance_type" {
    type = string
    description = "AWS EC2 Instance Type"
}

variable "aws_ec2_ami" {
    type = string
    description = "EC2 AMI for Amazon Linux 2"
}

main.tf

resource "aws_instance" "web" {
  instance_type = var.instance_type
  ami = var.aws_ec2_ami

  tags = {
    "Name" = "aws-ec2-demo"
  }
}

terraform.tfvars

aws_region = "us-east-1"
instance_type = "t2.micro"
aws_ec2_ami = "ami-04d29b6f966df1537"

We are done with the coding. Let us execute this Terraform code with the following commands:

# open your shell in the same project folder

# download the terraform core components 
# and initialize terraform in this directory
terraform init


# Validate changes to be made in AWS after the execution
terraform plan


# -auto-approve is used to skip manual approval prompt
terraform apply -auto-approve

Very Important

Clean Up

Do not forget to delete the infrastructure created to avoid incurring any costs.

# running this command will destroy all the resources
terraform destroy -auto-approve

Github Repo - https://github.com/g33kzone/tf-aws-ec2.git

In future posts, I will delve into Terraform Modules.

Configuring Jenkins on AWS EC2 Instance

Manish R Warang — Thu, 01 Jul 2021 19:26:54 +0000

This post provides an overview of the steps required to install and run Jenkins Server on Amazon Linux 2. By the end of the post, we will have a working Jenkins Server ready to configure pipelines.

Jenkins can be installed in numerous ways - Native system packages, Docker, standalone executable session. This post will be focused on native package installation.

Jenkins recommends the following minimum hardware requirements for Server configuration

1 CPU
256 MB of RAM
1 GB of drive space (although 10 GB is a recommended minimum if running Jenkins as a Docker container)

For Demo purposes, we can select the T2 micro instance type as it satisfies the above minimum criteria.

Prerequisites

AWS Cloud Account (Free tier will work)
A EC2 instance running with T2 micro for instance type
AWS key-pair to SSH into the EC2 instance
Configure firewalls (security groups) to allow SSH access into the EC2 instance

Installation Steps

Step 1. Initiate an SSH session into your Amazon Linux - EC2 instance.

Step 2. Update existing installed packages on the EC2 instance

sudo yum update -y

Step 3. Install most recent Open JDK (Java ver 11). Required for Jenkins installation

sudo amazon-linux-extras install java-openjdk11 -y

Step 4. Add Jenkins repo to Amazon Linux 2 server

sudo tee /etc/yum.repos.d/jenkins.repo<<EOF
[jenkins]
name=Jenkins
baseurl=http://pkg.jenkins.io/redhat
gpgcheck=0
EOF

Step 5. Import GPG Jenkins repo key

sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key

Step 6. Update list of repositories

sudo yum repolist

Step 7. Install Jenkins in Amazon Linux 2 server

sudo yum install jenkins -y

Step 8. Start Jenkins service

sudo systemctl start jenkins

Step 9. Enable Jenkins service to start at OS boot

sudo systemctl enable jenkins

Step 10. Confirm that the Jenkins Service is up and running

sudo systemctl status jenkins

Step 11. Validate if the Jenkins service is configured to autostart on system reboot.

sudo systemctl is-enabled jenkins

Step 12. Access Jenkins Server on EC2 instance
Jenkins service by default binds to port 8080. It will be accessible on

http://[server-ip-or-hostname]:8080

For the first time, Jenkins will prompt the Unlock Jenkins screen to authenticate the installation.

Step 13. Fetch password to Unlock Jenkins.
As advised the following command will provide the relevant password

cat /var/lib/jenkins/secrets/initialAdminPassword

Copy the automatically-generated alphanumeric password (between the 2 sets of asterisks). On the Unlock Jenkins page, paste this password into the Administrator password field and click Continue.

Step 14. Customize Jenkins
Recommended selecting Install suggested plugins to install the recommended set of plugins based on most common use cases.

The setup wizard shows the progression of Jenkins plugins being installed

Step 15. Create First Admin User
Enter relevant details to create Admin User and click ** Continue **

Step 16. Configure Valid DNS (if required)
Jenkins Instance access URL will be printed on the screen that follows. This can be changed with a valid DNS name. For the purpose of our demo, we will leave it unchanged and click ** Save and Finish **

Step 17. Jenkins is ready, Hurray!! You have now successfully configured Jenkins.

Jenkins Dashboard

Jenkins - Getting Started

Manish R Warang — Thu, 01 Jul 2021 19:23:11 +0000

Jenkins is an open-source automation server written in Java. It supports the entire software delivery lifecycle, including build, document, test, package, stage, deployment, static code analysis, and deployments via plugins. Several tools (e.g. unit tests, report generation, SCM, etc.) can be integrated with Jenkins via plugins. Jenkins has been the most widely adopted tool for CI/CD, thanks to its energetic, active community. This Jenkins community offers more than 1800+ plugins

The Jenkins project was started in 2004 (originally called Hudson) by Kohsuke Kawaguchi. Initially created for Continous Integration (CI), today Jenkins is capable to orchestrate the entire software delivery pipeline – called Continuous Delivery (CD) and even further to provide Continuous Deployment.

Following is a diagrammatic representation of the Jenkins pipeline.

In many scenarios, a single Jenkins server isn't sufficient to cater to the needs of multiple code-commits triggering zillions of pipelines. In such scenarios, distributed Jenkins architecture (Master Controller - Agents) can be leveraged.