DEV Community: Lucas de Camargo

AWS Governance with a Terragrunt Live Environment

Lucas de Camargo — Thu, 18 Dec 2025 12:59:39 +0000

Many tech enthusiasts have the desire to start or to move their stack to AWS and benefit from its 200+ global services. It may feel intuitive to create an account on AWS and start creating resources in the AWS Console, but this is the absolutely don't when working in the Cloud.

I have introduced some basic concepts of Governance on AWS in First steps to Governance on AWS using Terraform. To recap briefly: proper AWS governance is built on Three Governance Pillars: Organizations, Accounts, and Service Control Policies (SCPs). You may understand these concepts, but if you're still manually creating resources, you're missing the foundation that makes governance actually work: Infrastructure as Code.

Without IaC, you'll spend weeks trying to reverse-engineer your own infrastructure just to understand what you've built. Worse, when disaster strikes, like a misconfiguration, a security breach, or even accidental resource deletion, recovery becomes a manual, error-prone nightmare instead of a simple deployment operation.

If you're already familiar with Terraform, you know the pain: managing multiple AWS accounts leads to massive code duplication, enormous state files, and configurations you're afraid to touch because you've lost track of dependencies. This is where Terragrunt becomes essential. It eliminates code duplication through a hierarchical configuration system while maintaining clear separation of concerns across your multi-account environment.

In this guide, we'll build a Terragrunt live environment: a Git repository that versions your AWS infrastructure, organized by accounts and regions, with configurations that cascade intelligently through the hierarchy. We'll establish the foundation that makes deploying your governance resources (in the next post) both simple and maintainable.

Getting Started

The content of the Terragrunt Live Environment for this series is being maintained in my GitHub repository. In this post, I'm explaining the concepts of the repository structure, and deploying the first organization resource.

lucasdecamargo / terragrunt-live-example

Template for a Terragrunt stack using my AWS Terraform modules including Governance management.

terragrunt-live-example

Template for a Terragrunt stack using my AWS Terraform modules including Governance management.

View on GitHub

Installation

We'll be using three essential tools to manage our AWS infrastructure:

AWS CLI v2 authenticates and interacts with AWS services from your terminal. It's required for Terraform to communicate with AWS APIs.
Terraform is the Infrastructure as Code (IaC) engine that provisions and manages AWS resources using declarative configuration files.
Terragrunt is a thin wrapper around Terraform that helps eliminate code duplication and manage multiple environments.

Tip: If you're using VS Code, install these extensions for better HCL syntax support:

HashiCorp HCL

HashiCorp Terraform

Named Profiles

Go and quickly read my post about AWS Named Profiles, if you haven't yet.

At this point, it is considered that our AWS account doesn't yet have the SSO service enabled. Therefore, we need an access key that has the required permissions for deploying the resources we're creating in this post. These are the following:

Backend Policy for creating and updating the remote state files in our AWS root account.
Organization Policy for deploying the organization resources we're creating in this post.

Make sure to check out the permissions in my GitHub repository for the Terragrunt Live Environment example.

In your AWS Console, go to IAM -> Policies, and create a new policy for each of the JSON documents I have under the policies folder in my repository.
Next, go again to IAM -> Users, create a new terragrunt-root user, and attach these policies to it.
Generate a new pair o acess keys, and add it your AWS credentials file, as we discussed, with a name of acme-root, for example.

Live Environment

Our AWS infrastructure is going to live in a structured Git repository. Terragrunt allows us to define a folder structure based on our AWS accounts and to share variables and configurations across the stacks deployed to each account.

Typically, you'd see in Gruntwork's recommendation the following pattern for live repositories:

account
 └ _global
 └ region
    └ _global
    └ environment
       └ category
          └ resource

As an example:

backend
 └ _global
    └ networking
       └ route53-public-api-domain
 └ us-east-1
    └ _global
       └ storage
          └ s3-assets
    └ dev
       └ compute
          └ ecs-public-api-cluster

The _global folders contain resources that aren't tied to a specific region (like Route53 hosted zones, IAM roles, or S3 buckets with global names) or environment-agnostic resources.

However, when onboarding to the Cloud for the first time, it can be challenging to predict exactly what environments and categories you'll need from the start. We can simplify this structure significantly by encoding the environment into the account name itself:

account
 └ _global
 └ region
    └ resource

backend-dev
 └ _global
    └ route53-public-api-domain
 └ us-east-1
    └ s3-assets
    └ ecs-public-api-cluster

This approach aligns perfectly with our multi-account governance strategy from the previous post. We create a dedicated backend-dev account within our Development OU for all development work. When you're ready for production, you simply create a backend-prod account under your Production OU — each account maintains its own isolated folder structure.

This simplified pattern offers a better deal for startups because it is easier to get started with, as less folder nesting means less cognitive overhead when you're learning, and it is simple to evolve by refactoring it into more granular structure later as your needs grow.

Writing the Terragrunt Stack

Now that we understand the basics of a Terragrunt live environment, let's create our first stack. We'll build a Git repository to version our infrastructure state and walk through each component step by step.

We start by creating a Git repository locally that will version the state of our infrastructure. Consider these naming options:

terragrunt-live-management or terragrunt-live-governance - if you want separate repositories for different groups of stacks
terragrunt-live - if you're targeting a monorepo for all of your IaC stacks

I'm using terragrunt-live-example for this post, and our fictional organization will be called acme — we'll use this name when creating AWS resources.

Directory Structure

As we discussed earlier, the repository structure begins with a directory for an AWS account, followed by a directory for the region, and finally a directory for each resource we're deploying.

The key to understanding Terragrunt is that each layer uses an HCL file to declare local variables that cascade down through the hierarchy:

terragrunt-live-example
├── root                        # Root Account
│   ├── _global                 # Global Region
│   │   ├── organization        # Organization Resource
│   │   │   └── terragrunt.hcl
│   │   └── region.hcl
│   └── account.hcl
└── root.hcl

When deploying the organization resource in terragrunt.hcl, Terragrunt loads all definitions in this order: root.hcl → account.hcl → region.hcl → terragrunt.hcl. This layered approach keeps our configuration DRY (Don't Repeat Yourself) and makes it easy to share common settings across resources.

Locals

Terragrunt locals are similar to Terraform locals. They provide a mechanism to define named expressions within a terragrunt.hcl configuration file. These expressions are evaluated and their results can be referenced throughout the same configuration file, promoting reusability and reducing repetition.

In this example, we define the local variables for each directory layer.

# root/account.hcl
locals {
  account_name      = "root"
  aws_named_profile = "acme-root"

  tags = {
    "Account" = "root"
  }
}

# root/_global/region.hcl
locals {
  aws_region = "us-east-1"

  tags = {
    "Region" = "global"
  }
}

These locals are going to be available in the inner layers we are going to be working on, preventing us from repeating ourselves when declaring our Terraform modules based on these layer values.

Note that even though the region is _global, we still need to specify a default region (us-east-1) because the AWS CLI requires some region to be defined for API calls.

The Root File

The root.hcl file is the foundation of our Terragrunt configuration. We use Terragrunt's built-in functions read_terragrunt_config and find_in_parent_folders to automatically load the local variables we defined at each layer:

# root.hcl
locals {
  # Automatically load account-level variables
  account_vars = read_terragrunt_config(find_in_parent_folders("account.hcl"))
  # Automatically load region-level variables
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))
}

It's important to understand that find_in_parent_folders executes from the perspective of the resource level. This means when Terragrunt processes the organization resource, it searches upward through parent directories to find these configuration files.

Now we can use the inherited parameters to build new local variables:

# root.hcl
locals {
  # Automatically load account-level variables
  account_vars = read_terragrunt_config(find_in_parent_folders("account.hcl"))
  # Automatically load region-level variables
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))

  # Extract locals from the loaded configurations
  account_name      = local.account_vars.locals.account_name
  aws_named_profile = local.account_vars.locals.aws_named_profile
  aws_region        = local.region_vars.locals.aws_region

  # Define common variables used across all resources
  org_name = "acme"
  tags = {
    "ManagedBy" = "Terragrunt"
  }

  # Merge all tags hierarchically
  default_tags = merge(
    local.tags,
    local.account_vars.locals.tags,
    local.region_vars.locals.tags,
  )
}

We also declare an inputs block to pass variables to underlying Terraform modules, making them available for cross-stack references:

# root.hcl (continued)
inputs = merge(
  local.account_vars.locals,
  local.region_vars.locals,
)

Generating Providers

With our variables configured, we can now declare the AWS Provider for the account we're deploying to. If you're not familiar with providers, they are plugins that serve as the interface between Terraform and external services or platforms. They enable Terraform to manage resources, interact with APIs, and perform operations on various cloud platforms.

We use Terragrunt's generate block to inject the provider configuration into our Terraform modules. This keeps our configuration DRY by defining this common setup once in root.hcl rather than repeating it in every resource:

# root.hcl (continued)
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  profile = "${local.aws_named_profile}"
  region  = "${local.aws_region}"

  default_tags {
    tags = ${jsonencode(local.default_tags)}
  }
}
EOF
}

When you run terragrunt init, this automatically generates a provider.tf file in the Terraform working directory with the appropriate configuration for your environment.

Remote State Files

In IaC, the state file is crucial — it stores information about your infrastructure's current status, acting as a bridge between your configuration files and the actual resources deployed in AWS. Terraform uses this state file as its single source of truth to understand what infrastructure it has created and is managing. Every change plan that Terraform calculates is based on this state file.

Here's the key insight: since we're working in AWS, we can use our AWS accounts to store their own state files. It's also a best practice to use a separate state file per region for each AWS account. This approach:

Minimizes the potential impact of state file corruption
Improves performance by keeping state files smaller
Reduces cross-regional dependencies

We use Terragrunt's remote_state block to configure S3 as our state backend, with DynamoDB for state locking to prevent multiple Terraform operations from modifying the state concurrently:

# root.hcl (continued)
remote_state {
  backend = "s3"
  config = {
    encrypt        = true
    bucket         = "${local.org_name}-tfstate-${local.account_name}-${local.aws_region}-TRGRUNT"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = local.aws_region
    dynamodb_table = "tfstate-lock"
    profile        = local.aws_named_profile
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

This configuration will create an S3 bucket named acme-tfstate-root-us-east-1 when deploying the organization resource. Remember that S3 bucket names must be globally unique — if the name is already taken, you'll need to modify your naming convention by adding additional tokens like TRGRUNT.

Units (Resources)

A unit in Terragrunt is a directory containing a terragrunt.hcl file. This hermetic unit of infrastructure is the smallest deployable entity in Terragrunt. It’s also the most important feature Terragrunt has.

With our foundation complete, we can now declare AWS resources in our Terragrunt live stack. Let's use the organization module from the previous post in this series, which is maintained in my GitHub repository.

This module requires only the variable aws_allowed_regions which specifies what regions are allowed by our root organization:

# https://github.com/lucasdecamargo/terraform-aws-governance/tree/main/organization/variables.tf
variable "aws_allowed_regions" {
  description = "The AWS regions allowed by the organization."
  type        = list(string)
  default     = ["us-east-1", "us-east-2"]
}

To use this module, we create the simple Terragrunt configuration:

# root/_global/organization/terragrunt.hcl
terraform {
  source = "git::git@github.com:lucasdecamargo/terraform-aws-governance.git//organization"
}

include "root" {
  path = find_in_parent_folders("root.hcl")
}

inputs = {
  aws_allowed_regions = ["us-east-1", "us-east-2", "sa-east-1"]
}

Let's break down what each block does:

The terraform block specifies the source of the module we're deploying, pointing to the organization directory in my GitHub repository (by default, it uses the main branch)
The include block links this resource to our live repository configuration, starting with root.hcl
The inputs block provides the required variables to the Terraform module

Deploying with Terragrunt

At this point, our Terragrunt live environment is configured and ready to deploy. Let's understand how Terragrunt's deployment workflow works and the different ways you can deploy your infrastructure.

Unlike plain Terraform, Terragrunt automatically calls terraform init before other commands when it detects that initialization is needed. After creating creating the remote state backend, the typical workflow is simply:

Plan: Review what changes will be made
Apply: Execute the changes

Once the backend is bootstrapped for an account, Terragrunt handles initialization automatically for subsequent commands and no manual init step required unless you need to update providers or modules.

Deploying the Remote State Backend

Before deploying any infrastructure in an account, you need to create the remote state backend. Our root.hcl configuration specifies an S3 bucket and DynamoDB table for state management at the account level, but these don't exist yet.

You must explicitly bootstrap these backend resources using:

cd root/_global/organization
terragrunt init --backend-bootstrap

This creates the backend infrastructure for the entire root account:

An S3 bucket named acme-tfstate-root-us-east-1-TRGRUNT (with encryption enabled)
A DynamoDB table named tfstate-lock for state locking

Note: Automatic backend bootstrapping is deprecated functionality and will not be the default behavior in future Terragrunt versions. Always use the --backend-bootstrap flag explicitly.

Once the backend is created for an account, you don't need to bootstrap it again—all units within that account will use the same S3 bucket and DynamoDB table for their state files.

Deployment Scopes

You can run run --all commands at any depth of the stack to run the units in that stack and all of its children. Here are the different scopes:

Deploying a Single Unit*

The most common approach — deploy one specific resource:

cd root/_global/organization
terragrunt plan
terragrunt apply

Deploying an Entire Region

Deploy all units within a region:

cd root/_global
terragrunt run --all plan
terragrunt run --all apply

Deploying an Entire Account

Deploy all resources across all regions in an account:

cd root
terragrunt run --all plan
terragrunt run --all apply

When using run --all, Terragrunt analyzes the dependencies in your stack and determines an order for runs so that outputs are ready to be used as inputs in dependent units.

Important: When using run --all with apply or destroy, Terragrunt automatically adds the -auto-approve flag, meaning you won't be prompted to confirm each unit. Always run plan first to review changes.

What's Next?

We've established our Terragrunt live environment with the complete repository structure, configuration hierarchy, and deployment workflow. Our infrastructure is now version-controlled, DRY, and ready to scale across multiple accounts and regions.

Follow me to get updates about my series IaC Startup on AWS. I'm writing new posts about configuring AWS named profiles, and deploying governance infrastructure with Terragrunt and Terraform. By the end, you'll have a production-ready, multi-account AWS environment managed entirely through Infrastructure as Code.

Also, make sure to check my GitHub profile for new projects and updates.

Buy me a coffee if you like this post! :)

Configuring AWS Named Profiles

Lucas de Camargo — Sat, 13 Dec 2025 21:34:49 +0000

Following up with our series of engineering an IaC Startup on AWS, there's this one addendum I must talk about.

When working with multiple AWS accounts you need a way to tell the AWS CLI and Terraform which account to use for each operation. This is where named profiles come in.

A named profile is a collection of credentials and configuration settings that represents a specific identity in a specific AWS account. Instead of constantly switching credentials or passing them as command-line arguments, you define profiles once in your local AWS configuration files and reference them by name.

For example, you might have:

management-admin - Administrator access to your management account
backend-dev-admin - Administrator access to your development account
backend-prod-developer - Limited developer access to production

Named profiles are essential when working with Terraform and Terragrunt. Each stack may need to deploy resources to a different AWS account or assume different roles. By using named profiles in your Terraform provider configuration, you explicitly declare which identity should be used:

provider "aws" {
  region  = "us-east-1"
  profile = "backend-dev-admin"
}

When Terraform runs, it looks for this profile in your local AWS configuration files and authenticates accordingly.

AWS profiles are configured in two files in your home directory:

~/.aws/credentials - Stores authentication credentials (access keys or SSO configuration)

~/.aws/config - Stores regional and output preferences

There are two primary ways to authenticate a profile.

Single Sign-On (SSO) - Recommended

This is the secure, modern approach that aligns with AWS best practices and the governance foundation we established. Instead of long-lived access keys, SSO provides temporary credentials that automatically expire:

# ~/.aws/credentials
[profile management-admin]
sso_start_url = https://mycorp.awsapps.com/start
sso_region = us-east-1
sso_account_id = 123456789012
sso_role_name = AdministratorAccess

It is recommended to explicitly define the configuration values for each profile:

# ~/.aws/config
[profile management-admin]
region = us-east-1
output = json

To use an SSO profile, you need to login with the AWS CLI before running Terraform or any CLI command, which opens a browser for authentication and caches temporary credentials.

aws sso login --profile management-admin

# Test it with
aws sts get-caller-identity --profile management-admin

Access Keys

Long-lived access keys can be stored directly in ~/.aws/credentials, but this approach is discouraged by AWS security best practices because:

Keys don't expire automatically
If accidentally committed to Git or exposed, they remain valid until manually rotated
They don't provide the same audit trail as SSO

# ~/.aws/credentials (avoid this approach if possible)
[management-admin]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Important: The default profile (with no profile prefix) is the profile used when no specific profile is specified. For safety, leave it unconfigured.

Testing Profiles

Once you've configured your profiles, it's essential to verify they're working correctly before attempting to deploy infrastructure.

For SSO profiles, start by logging in:

# Login to the SSO profile
aws sso login --profile management-admin

This opens your browser for authentication. After successfully logging in, the credentials are cached locally (typically for 8-12 hours, depending on your SSO configuration).

For access key profiles, the test is simpler since no login is required.

Test your profiles by retrieving your caller identity:

aws sts get-caller-identity --profile management-admin

You should see output like:

{
    "UserId": "AROA...:user@example.com",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/AdministratorAccess/user@example.com"
}

You can also list resources to verify permissions:

# List S3 buckets
aws s3 ls --profile management-admin

# List EC2 instances in a specific region
aws ec2 describe-instances --region us-east-1 --profile management-admin

If you see "UnauthorizedOperation" or "AccessDenied" errors, check that your IAM user or role has the necessary permissions attached.

Deploying Governance before SSO

Here's the chicken and the egg! The problem with the first and recommended approach is that we need to first enable SSO in AWS and define an account for each user with the right accesss permissions. Without SSO, we're left with no choice rather using a temporary access key to work with IaC on AWS.

If creating an admin user in IAM with no restrictions feels too much and not a good governance approach for you, I have written a set of granular permissions that are needed for deploying our organization resources until we configure SSO.

Check out the permissions in my GitHub repository for the Terragrunt Live Environment example.

In your AWS Console, go to IAM -> Policies, and create a new policy for each of the JSON documents I have under the policies folder in my repository.
Next, go again to IAM -> Users, create a new terragrunt-root user, and attach these policies to it.
Generate a new pair o acess keys, and add it your AWS credentials file, as we discussed, with some name of acme-root, for example.

What's Next?

Now that we've configured AWS named profiles for secure authentication across multiple accounts, we have everything we need to actually deploy our governance infrastructure.

Follow me to get updates about my series IaC Startup on AWS. I'm writing new posts about deploying governance infrastructure with Terragrunt and Terraform.

Also, make sure to check my GitHub profile for new projects and updates.

Buy me a coffee if you like this post! :)

Introducing the Three AWS Governance Pillars with Terraform

Lucas de Camargo — Sat, 13 Dec 2025 20:30:30 +0000

I've seen many excellent tech enthusiasts come up with great ideas to start a service, an app, or even an entire business. Most are familiar with what AWS offers, but few understand how to establish proper governance in their new organization, no matter how small. This gap often leads to painful consequences down the road — infrastructure that's difficult to audit, security vulnerabilities that grow with each new resource, and technical debt that becomes exponentially harder to fix as your startup scales.

Although some prefer to use backend as a service tools such as Supabase for hosting their website and building their user base, these tools not only come with high costs, but they're limited frameworks when compared to the 200+ global, on-demand services offered by AWS. More importantly, they abstract away the infrastructure knowledge that becomes critical as you grow.

On the other hand, starting up with AWS may feel overwhelming at first glance, not to mention that management and governance may get out of control when creating resources directly in the AWS Console. Without proper structure from day one, you'll find yourself manually clicking through the console, creating resources inconsistently across environments, with no clear audit trail of who changed what or when. I've seen startups spend weeks trying to reverse-engineer their own infrastructure just to understand what they've built.

The good news? Once you learn how to get started on AWS the right way—with Infrastructure as Code and proper governance from the beginning—it becomes intuitive to scale at very low prices, like $0.50 per month for hosting a website. More importantly, you'll have a reproducible, auditable, and secure foundation aligned with the AWS Well-Architected Framework. Every new project, environment, or team member can leverage the same proven patterns, eliminating weeks of setup time and preventing costly security mistakes before they happen.

This series will show you exactly how to build that foundation, starting with the governance primitives that AWS itself recommends: Organizations, Accounts, and Policies. Whether you're a solo founder or building with a small team, these patterns will save you countless hours and headaches as you grow.

The Three Governance Pillars

To get hands-on with your new infrastructure skills, let me introduce three AWS resources that are going to be the ground basis of our business: Organizations, Accounts, and Policies.

AWS Organizations and Organization Units

Like the control tower for your entire cloud infrastructure, an Organization provides centralized management for multiple AWS accounts, allowing you to structure them hierarchically like a family tree. At the very top sits the Organization Root - this is the parent container for everything in your AWS account. Below the root, you create Organizational Units (OUs), which are logical groupings that help you organize accounts by function, environment, or business unit. For example, you might have separate OUs for "Production," "Development," and "Security." The key difference is that the Organization Root is singular and encompasses everything, while OUs are the building blocks you create to segment and organize your accounts.

Organizations work hand-in-hand with Accounts (which live inside OUs) and Policies (which can be attached at any level of the hierarchy to enforce governance rules).

AWS Organization Accounts

Individual AWS accounts are the fundamental security and resource boundaries in AWS. Each account acts as an isolated container for resources, with its own billing, access controls, and resource limits. Working with multiple accounts isn't just a nice-to-have - it's a critical best practice because it provides natural blast radius containment (if something goes wrong in one account, others remain unaffected), simplified compliance boundaries, and clearer cost allocation.

Within an Organization, accounts nest inside units and inherit any policies applied to their parent OU or the organization root, creating a powerful governance cascade from top to bottom.

Service Control Policies (SCPs)

Organization policies are your guardrails - they define the maximum permissions that any entity within an account can have, regardless of what their IAM permissions say. Think of them as a safety net that prevents accidental or intentional actions that could harm your organization. Common examples include preventing accounts from leaving the organization, restricting which AWS regions can be used, or blocking the deletion of critical resources like CloudTrail logs.

SCPs attach to the Organization Root, OUs, or individual accounts, and they flow downward through the hierarchy - a policy attached to an OU automatically applies to all accounts within that OU and any nested OUs below it.

The example above illustrates a simple but mature organizational structure. AWS Organizations is designed to evolve with your startup - you'll typically begin with just two or three accounts addressing immediate needs, then add new organizational units and accounts as real requirements emerge from your business operations. This iterative approach isn't a compromise; it's the recommended path because it ensures every piece of your infrastructure exists for a concrete purpose rather than speculative needs. Since you can freely reorganize accounts and OUs later, there's no penalty for starting simple.

Terraform Modules

In this section, we'll create reusable Terraform modules for our governance resources. These modules will live in a GitHub repository called terraform-aws-governance and will be designed to work together as building blocks for your AWS foundation.

Choosing the Right Granularity

When structuring Infrastructure as Code (IaC), there are three common approaches, each with trade-offs:

Monolithic: One large Terraform configuration managing everything. Quick to start but becomes cubersome as you grow and risky to change.
One-repo-per-module: Each module in its own repository. Maximum isolation but harder to maintain consistency and versioning across related modules.
Grouped modules: Related modules in a single repository (our choice). This balances reusability with maintainability - our terraform-aws-governance repo will contain modules for organizations, organizational units, accounts, and eventually billing and audit configurations.

We're choosing the grouped approach because governance resources are inherently related and often change together. This structure also works excellently with Terragrunt, which we'll use later to orchestrate these modules into a complete infrastructure stack while maintaining clear separation of concerns.

Starting with the essentials, we'll create three foundational modules:

organization: Creates the AWS Organization with baseline security policies
organizational-unit: Creates OUs with customizable policies
account: Provisions new AWS accounts within specified OUs

Each module is designed to be stateless and idempotent, meaning you can safely run them multiple times without side effects. The folder structure is defined as below:

terraform-aws-governance/
├── organization/
│   ├── main.tf
│   ├── variables.tf
│   └── outputs.tf
├── organization-unit/
│   ├── main.tf
│   ├── variables.tf
│   └── outputs.tf
└── account/
    ├── main.tf
    ├── variables.tf
    └── outputs.tf

The full implementation of this blog post is available in my GitHub repository.

lucasdecamargo / terraform-aws-governance

Terraform modules for AWS governance services, like Organizations, Identity and Budget.

Terraform AWS Governance Modules

A collection of Terraform modules that codify the governance pillars featured in the blog post IaC Startup on AWS: Governance Pillars.¹

The modules help you bootstrap an AWS Organization with opinionated Service Control Policies (SCPs), create Organizational Units (OUs), and provision member accounts with consistent guardrails. They are designed to be composed together or orchestrated with tools such as Terragrunt.

Module Overview

organization: Creates the AWS Organization, enables key services, and attaches baseline SCPs that lock operations to approved regions, protect logging, and keep IAM Identity Center centralized.
organization-unit: Provisions an OU under any parent and allows you to attach existing SCPs or create a custom policy for that branch of the hierarchy.
account: Automates new AWS member accounts with optional billing console access, inherited SCP attachments, and per-account guardrails.

organization

Enables IAM and IAM Identity Center integration, sets the feature…

View on GitHub

Let's begin with the organization module, which establishes our governance foundation.

Organization

This module creates the foundation of your AWS governance structure. Let's start with the core resource - the AWS Organization itself:

resource "aws_organizations_organization" "this" {
  aws_service_access_principals = [
    # Allows IAM roles to work across accounts
    "iam.amazonaws.com",
    # Enables centralized user management
    "sso.amazonaws.com"
  ]

  # Enables all organization features
  feature_set = "ALL"

  enabled_policy_types = [
    # Primary governance tool (SCPs)
    "SERVICE_CONTROL_POLICY"
  ]
}

This resource can only be created once, from what will become your management account. The feature_set = "ALL" is crucial - it enables Service Control Policies, which are your primary governance tool. Without this, you'd only get consolidated billing.

Service Control Policies (SCPs) are preventive controls that define the maximum permissions for your organization. Even if someone has full administrator access in an account, they can't perform actions that SCPs deny. Let's implement three essential policies that every startup should have from day one.

SCP 1: Restricting AWS Regions

Many startups need to restrict where data can be stored for compliance or cost reasons. This policy ensures resources can only be created in your approved regions:

variable "aws_allowed_regions" {
  description = "The AWS regions allowed by the organization."
  type        = list(string)
  default     = ["us-east-1", "us-east-2"]
}

resource "aws_organizations_policy" "deny_non_allowed_regions" {
  name = "DenyNonAllowedRegions"
  type = "SERVICE_CONTROL_POLICY"

  # Ensure the policy is created after the organization is created
  depends_on = [aws_organizations_organization.this]

  content = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid      = "DenyAllOutsideAllowedRegions"
      Effect   = "Deny"
      Action   = "*"
      Resource = "*"
      Condition = {
        StringNotEquals = {
          "aws:RequestedRegion" = var.aws_allowed_regions
        }
      }
    }]
  })
}

resource "aws_organizations_policy_attachment" "deny_non_allowed_regions" {
  policy_id = aws_organizations_policy.deny_non_allowed_regions.id
  target_id = aws_organizations_organization.this.roots[0].id
}

Notice how we use a variable for aws_allowed_regions. This makes it easy to adjust as your compliance requirements evolve. The region us-east-1 is always needed for AWS global resources that are hosted in the North Virginia region. If you are in western Europe, you'd use ["us-east-1", "eu-west-1"].

In this first deployment, we need to tell Terraform that the policies we are creating depend on the organization resource with the argument depends_on, otherwise it will try to create the policies before the organization completes, which causes an error.

The roots[0].id refers to the organization's root - the top of your organizational hierarchy. Policies attached here cascade down to every OU and account below.

SCP 2: Centralized Authentication

This policy prevents individual accounts from creating their own IAM Identity Center instances (Accounts), ensuring authentication stays centralized:

resource "aws_organizations_policy" "deny_member_account_sso" {
  name = "DenyMemberAccountSSO"
  type = "SERVICE_CONTROL_POLICY"

  # Ensure the policy is created after the organization is created
  depends_on = [aws_organizations_organization.this]

  content = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Deny"
      Action = [
        "sso:CreateInstance",
        "sso:DeleteInstance"
      ]
      Resource = "*"
    }]
  })
}

resource "aws_organizations_policy_attachment" "deny_member_account_sso" {
  policy_id = aws_organizations_policy.deny_member_account_sso.id
  target_id = aws_organizations_organization.this.roots[0].id
}

Without this policy, any account could set up its own identity provider, leading to a fragmented authentication landscape that's impossible to audit or secure properly.

SCP 3: Protecting Critical Resources

This policy prevents accidental or malicious deletion of your audit trail:

resource "aws_organizations_policy" "protect_organization_resources" {
  name        = "ProtectOrganizationResources"
  type        = "SERVICE_CONTROL_POLICY"

  # Ensure the policy is created after the organization is created
  depends_on = [aws_organizations_organization.this]

  content = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "ProtectCloudTrail"
        Effect = "Deny"
        Action = [
          "cloudtrail:DeleteTrail",
          "cloudtrail:StopLogging"
        ]
        Resource = "*"
      },
      {
        Sid    = "PreventLeavingOrganization"
        Effect = "Deny"
        Action = [
          "organizations:LeaveOrganization"
        ]
        Resource = "*"
      }
    ]
  })
}

resource "aws_organizations_policy_attachment" "protect_organization_resources" {
  policy_id = aws_organizations_policy.protect_organization_resources.id
  target_id = aws_organizations_organization.this.roots[0].id
}

These protections ensure that even if credentials are compromised, attackers can't cover their tracks by deleting logs or remove accounts from your organization's oversight.

Module Outputs

Finally, we export key information that other modules will need:

output "organization_root_id" {
  value       = aws_organizations_organization.this.roots[0].id
  description = "The ID of the root of the AWS Organization"
}

output "scp_policy_ids" {
  value = {
    deny_non_allowed_regions       = aws_organizations_policy.deny_non_allowed_regions.id
    protect_organization_resources = aws_organizations_policy.protect_organization_resources.id
    deny_member_account_sso        = aws_organizations_policy.deny_member_account_sso.id
  }
  description = "Map of Service Control Policy IDs"
}

These outputs become the connection points for your next modules. The organization_root_id is where you'll attach OUs, and the scp_policy_ids map lets other modules reference these policies without hardcoding IDs.

Remember that this module must be run from your management account and can only be applied once. In practice, you'll run this before creating any other infrastructure. The account running this Terraform code becomes the permanent management account for your organization - choose wisely, as this can't be changed later without recreating the entire organization.

Organization Unit

Now that we have our Organization foundation, we need a way to create logical groupings for our accounts. Organizational Units (OUs) are containers that help you organize accounts based on their function, compliance requirements, or operational needs.

The core of our OU module is surprisingly simple - creating an OU requires just a name and a parent:

variable "name" {
  description = "Name of the Organizational Unit"
  type        = string

  validation {
    condition     = length(var.name) <= 128
    error_message = "OU name must be 128 characters or less"
  }
}

variable "parent_id" {
  description = "ID of the parent OU or Organization Root"
  type        = string
}

resource "aws_organizations_organizational_unit" "this" {
  name      = var.name
  parent_id = var.parent_id
}

The parent_id is crucial here - it determines where in your organization hierarchy this OU sits. This could be the organization root (for top-level OUs like "Production" or "Development") or another OU (for nested structures like "Production/EU" or "Production/US").

What makes OUs powerful is their ability to have their own policies while inheriting from their parents. Our module supports two approaches to policy management.

Often, you'll want to reuse policies across multiple OUs. For example, you might have a standard "DevelopmentRestrictions" policy that applies to all development OUs:

variable "attach_policy_ids" {
  description = "List of existing SCP policy IDs to attach to this OU"
  type        = list(string)
  default     = []
}

resource "aws_organizations_policy_attachment" "existing" {
  for_each = toset(var.attach_policy_ids)

  policy_id = each.value
  target_id = aws_organizations_organizational_unit.this.id
}

Using for_each here allows you to attach multiple policies in a single module call. The toset() function ensures Terraform treats the policy IDs as unique identifiers rather than an ordered list, which prevents unnecessary recreations if the order changes.

Creating OU-Specific Policies

Sometimes an OU needs its own specific restrictions. Our module allows creating a custom policy alongside the OU:

variable "create_policy" {
  description = "Optional custom SCP policy to create and attach to this OU"
  type = object({
    name        = string
    description = string
    content     = string
  })
  default = null
}

resource "aws_organizations_policy" "custom" {
  count = var.create_policy != null ? 1 : 0

  name        = var.create_policy.name
  description = var.create_policy.description
  type        = "SERVICE_CONTROL_POLICY"
  content     = var.create_policy.content
}

The create_policy variable groups related attributes together and makes the module interface cleaner. When calling the module, you either provide all policy details or nothing - preventing partial configurations that could cause errors.

The count parameter makes this optional - the policy is only created if you provide the create_policy variable. This pattern keeps the module flexible without requiring complex policy definitions for simple OUs.

Module Outputs

The outputs are designed to support module composition - using this module's outputs as inputs to others:

output "id" {
  value       = aws_organizations_organizational_unit.this.id
  description = "The ID of the Organizational Unit"
}

output "arn" {
  value       = aws_organizations_organizational_unit.this.arn
  description = "The ARN of the Organizational Unit"
}

output "name" {
  value       = aws_organizations_organizational_unit.this.name
  description = "The name of the Organizational Unit"
}

output "parent_id" {
  value       = aws_organizations_organizational_unit.this.parent_id
  description = "The ID of the parent OU or Organization Root"
}

output "custom_policy_id" {
  value       = var.create_policy != null ? aws_organizations_policy.custom[0].id : null
  description = "The ID of the custom policy created for this OU (if any)"
}

The id output becomes the parent_id input when creating nested OUs or the parent_id when creating accounts. The custom_policy_id makes it possible to re-use the same custom policy in other resources.

Account

With our organization and OUs in place, the next step is to create individual AWS accounts that will live under this governance structure.
Each account represents an isolated environment — for example, development, staging, or production. The goal of this module is to make account creation repeatable, compliant, and safe.

Creating accounts programmatically through AWS Organizations can be tricky: accounts are immutable in many ways and must always have a unique email address. Having a module for Accounts allow us to wrap this process with sensible defaults and governance best practices.

The main resource for provisioning new accounts is aws_organizations_account. This resource instructs AWS Organizations to create a new member account within your organization hierarchy:

variable "name" {
  description = "Name for the account to be created"
}

variable "email" {
  description = "Email address for this account, needs to be unique"
}

variable "billing_access" {
  description = "Whether the account should have access to the AWS Billing Console"
  type        = bool
  default     = false
}

resource "aws_organizations_account" "account" {
  name                       = var.name
  email                      = var.email
  iam_user_access_to_billing = var.billing_access ? "ALLOW" : "DENY"

  lifecycle {
    # Ignore changes to prevent the account from being deleted and recreated
    ignore_changes = [name, email]
  }
}

The parameter iam_user_access_to_billing controls whether IAM users can access the Billing Console in that account. By default, it’s disabled (DENY), which aligns with AWS’s security best practices.

Once an AWS account is created, its name and email can’t be modified via Terraform — trying to do so would trigger a full recreation of the account (which is not allowed). Ignoring these attributes with ignore_changes prevents unnecessary drift and protects against accidental destruction of live accounts.

Just like OUs, accounts inherit policies from their parents but can also have additional Service Control Policies (SCPs) applied directly. You can pass one or more existing policy IDs — for example, those defined at the organization or OU level — and they’ll automatically attach to this account:

variable "attach_policy_ids" {
  description = "List of existing SCP policy IDs to attach to this account"
  type        = list(string)
  default     = []
}

resource "aws_organizations_policy_attachment" "existing" {
  for_each = toset(var.attach_policy_ids)

  policy_id = each.value
  target_id = aws_organizations_account.account.id
}

The same way, if a specific account requires a unique restriction — for instance, limiting IAM role creation or disabling certain AWS services — you can create and attach a custom SCP directly:

variable "create_policy" {
  description = "Optional custom SCP policy to create and attach to this account"
  type = object({
    name        = string
    description = string
    content     = string
  })
  default = null
}

resource "aws_organizations_policy" "custom" {
  count = var.create_policy != null ? 1 : 0

  name        = var.create_policy.name
  description = var.create_policy.description
  type        = "SERVICE_CONTROL_POLICY"
  content     = var.create_policy.content
}

resource "aws_organizations_policy_attachment" "custom" {
  count = var.create_policy != null ? 1 : 0

  policy_id = aws_organizations_policy.custom[0].id
  target_id = aws_organizations_account.account.id
}

The count condition makes this optional — the policy and its attachment are only created when you pass a create_policy object. This pattern gives the module flexibility while keeping the configuration concise.

Module Outputs

The module exposes a few key outputs to integrate with the rest of your infrastructure:

output "account_id" {
  value       = aws_organizations_account.account.id
  description = "ID of the account created"
}

output "email" {
  value       = var.email
  description = "Email of account"
}

output "custom_policy_id" {
  value       = var.create_policy != null ? aws_organizations_policy.custom[0].id : null
  description = "The ID of the custom policy created for this account (if any)"
}

output "all_attached_policy_ids" {
  value = concat(
    var.attach_policy_ids,
    var.create_policy != null ? [aws_organizations_policy.custom[0].id] : []
  )
  description = "List of all policy IDs attached to this account"
}

These outputs allow other modules — such as those that handle IAM Identity Center assignments, logging, or baseline configurations — to reference the account and its governance settings programmatically.

What's Next?

We've built the foundational Terraform modules for AWS governance — Organizations, Organizational Units, and Accounts. Now we need to deploy them.

Follow me to get updates about my series IaC Startup on AWS. I'm writing new posts about configuring AWS named profiles, and deploying governance infrastructure with Terragrunt and Terraform.

Also, make sure to check my GitHub profile for new projects and updates.

Buy me a coffee if you like this post! :)

Go Beyond Viper and Cobra: Declarative Field-Driven Configuration for Go Apps

Lucas de Camargo — Sat, 23 Aug 2025 02:12:57 +0000

Production Go applications constantly require the introduction of new configuration parameters. Based on the Open-Closed Principle, once we define a strategy for managing configuration fields, introducing new values becomes only small extensions. In this article, I'm proposing the definition of a Field structure for declaring configuration settings that are easily integrated with CLI completions and documentation generation.

A template is available in my Github Repository, and it uses three packages:

Viper: A complete configuration solution for Go applications that handles multiple file formats (YAML, JSON, TOML, HCL, ENV), environment variables, and provides a unified interface for accessing configuration values. Viper acts as a registry for all your application's configuration needs, with automatic environment variable binding and a precedence system for value resolution.
Cobra: A powerful CLI framework that provides a simple interface to create modern command-line applications with sophisticated features like nested subcommands, POSIX-compliant flags, automatic help generation, and shell completions. It's the same framework used by Kubernetes, Docker, and GitHub CLI for building their command-line interfaces.
Go Validator: A struct and field validation library that enables validation through struct tags, custom validation functions, and cross-field validation. It provides a declarative way to define validation rules directly in your struct definitions, making it easy to ensure data integrity throughout your application.

Requirements
- Functionalities
- The App Example
- Application Fields
- Network Fields
Implementation
- File Structure
  - Constants
- Field-Driven Configuration
- The Field Structure
- Field Collection
- Real Field Definitions
  - Application Fields
  - Network Fields
- Validators
- Configs Powered by Viper
- Writing the CLI only once with Cobra
- The Root Command
  - Binding Flags
- The Config Commands
  - Generating Completions
  - Listing Configuration Values
  - Setting Configuration Values
  - Auto-Documentation: Describing Parameters
Results
- Building
- Usage
Conclusion
- Extending the Solution
- Buy me a coffee

Requirements

Modern production applications need robust configuration management that can adapt to different environments, validate inputs, and provide clear documentation to users. The approach presented here addresses these needs by creating a unified system where configuration metadata lives alongside the configuration values themselves. This creates a single source of truth that eliminates duplication and reduces the chance of documentation drift.

Functionalities

Single Source of Truth: Configuration fields are defined once with all metadata (validation, documentation, defaults); and can be easily extended.

Default Value Definition with Build Flags: Production apps are often built for different environments, therefore some default configuration values must be defined by Go ldflags.

Multiple Formats: Viper support for YAML, JSON, TOML, HCL, and ENV configuration files.

Auto CLI Integration: Seamless integration with Cobra for command-line interfaces, data validation, and auto-completion.

Type Safety: Strongly typed configuration with validation by Go Validator tags, custom functions, and literals.

Environment Variables: Automatic binding with configurable prefix.

Documentation: Built-in help and documentation generation.

The App Example

Our application is called confapp, and it uses two groups of configuration: application base parameters, like logging and updates, and network configuration, like proxies. The architecture demonstrates how to organize configuration fields into logical groups, making it easy for users to understand and manage related settings together. Each field carries its complete metadata, ensuring that validation rules, documentation, and defaults are always consistent across the entire application.

Users are expected to use the CLI to configure the application, like:

confapp config set --log.level debug --update.auto true

Application Fields

Field	Type	Default	Description
`environment`	string	`dev`	Application environment (hidden)
`log.level`	string	`info`	Logging level (debug, info, warn, error)
`log.output`	string	`nil`	Log output file path
`update.auto`	bool	`false`	Enable auto-updates
`update.period`	duration	`15m`	Update check period

Network Fields

Field	Type	Default	Description
`proxy.http`	string	`nil`	HTTP proxy
`proxy.https`	string	`nil`	HTTPS proxy

Implementation

The implementation follows a modular approach where each component has a specific responsibility. The configuration module defines the fields and their metadata, Viper handles the actual storage and retrieval of values, and Cobra provides the user interface. This separation of concerns makes the system maintainable and allows each component to evolve independently while maintaining a stable interface between them.

File Structure

The project structure reflects the separation between CLI commands and configuration logic. The cmd directory contains all CLI-related code, while the internal/config directory houses the configuration management core. This organization makes it clear where to find specific functionality and ensures that the configuration logic remains independent of the CLI implementation.

go-appconfig-example/
├── cmd/                   # CLI command implementations
│   ├── root.go            # Root command and global flags
│   └── config.go          # Configuration management commands
├── internal/
│   ├── config/            # Configuration management core
│   │   ├── fields.go      # Field definitions and collections
│   │   ├── config.go      # Viper integration
│   │   └── validators.go  # Custom validation functions
│   └── consts/            # Application constants
│       └── consts.go      # Go flags like app name, version, etc.
├── main.go                # Application entry point
└── go.mod                 # Go module definition

Constants

Application constants define global values that remain consistent throughout the application's lifecycle. These can be overridden at build time using Go's -ldflags feature, allowing you to customize the application name, version, or other constants without modifying the source code. This is particularly useful for CI/CD pipelines where different builds might need different configurations.

// internal/consts/consts.go

var (
  // Application parameters
  AppName    = "confapp"
  AppVersion = "v0.1.0"

  // Config parameters
  ConfigEnvPrefix = "CONFAPP"
)

Field-Driven Configuration

Field-Driven Configuration is a design pattern where configuration parameters are defined as structured data containing all their metadata. Instead of scattering validation rules, documentation, and default values across different parts of the codebase, each field becomes a self-contained unit that describes everything about a configuration parameter. This approach ensures consistency and makes it trivial to add new configuration options without touching multiple files.

The Field Structure

The Field structure is the cornerstone of our configuration system. It encapsulates not just the value of a configuration parameter, but also its type, validation rules, documentation, and any other metadata needed to work with that parameter. This rich metadata enables automatic generation of CLI flags, validation logic, and documentation, all from a single definition.

// internal/config/fields.go

type FieldType string

const (
  FieldTypeString   FieldType = "string"
  FieldTypeBool     FieldType = "bool"
  FieldTypeInt      FieldType = "int"
  FieldTypeFloat    FieldType = "float"
  FieldTypeDuration FieldType = "duration"
)

type Field struct {
  Name         string            // Configuration key: "log.level"
  Group        string            // Logical grouping: "logging"
  Type         FieldType         // Data type: string, bool, duration
  Default      any               // Default value
  Description  string            // CLI help text
  Docstring    string            // Detailed documentation
  ValidValues  []any             // Allowed values: ["debug", "info", "warn"]
  ValidateTag  string            // Go validator tag: "required,oneof=debug info warn"
  ValidateFunc func(any) error   // Custom validation function
  Example      string            // Usage example
  Hidden       bool              // Hide from standard listings
  Deprecated   string            // Deprecation message if field is deprecated
}

// Validate performs validation on a field value using the configured validation rules.
// Returns nil if validation passes, or an error describing the validation failure.
func (f *Field) Validate(value any) error {
  if value == nil {
    return nil
  }

  // Check against valid values if specified
  if f.ValidValues != nil {
    if slices.Contains(f.ValidValues, value) {
      return nil
    }
    return fmt.Errorf("%s: valid values: %v", f.Name, f.ValidValues)
  }

  // Apply validator tag if specified
  if f.ValidateTag != "" {
    if err := validator.New().Var(value, f.ValidateTag); err != nil {
      return fmt.Errorf("%s: %w", f.Name, err)
    }
  }

  // Apply custom validation function if specified
  if f.ValidateFunc != nil {
    if err := f.ValidateFunc(value); err != nil {
      return fmt.Errorf("%s: %w", f.Name, err)
    }
  }

  return nil
}

Field Collection

The FieldCollection acts as a registry for all configuration fields in your application. It provides methods to add new fields dynamically and retrieve them efficiently. This centralized collection ensures that all parts of the application work with the same field definitions, maintaining consistency across CLI commands, validation, and documentation generation.

// internal/config/fields.go

// Fields is the global collection of all configuration fields
var Fields = FieldCollection{}

type FieldCollection []*Field

func (fc *FieldCollection) Add(fields ...*Field) {
  for _, field := range fields {
    // Check if field already exists and replace it
    for i, existingField := range *fc {
      if existingField.Name == field.Name {
        (*fc)[i] = field
        goto nextField
      }
    }
    // Field doesn't exist, append it
    *fc = append(*fc, field)
  nextField:
  }
}

// Map returns a map for O(1) field lookup by name
func (fc *FieldCollection) Map() FieldMap {
  fields := make(FieldMap)
  for _, f := range *fc {
    fields[f.Name] = f
  }
  return fields
}

Real Field Definitions

With the Field structure and FieldCollection in place, we can now define actual configuration parameters. Each field definition becomes a single source of truth that contains everything needed to work with that configuration value. The use of init() functions ensures that fields are automatically registered when the package is imported, eliminating the need for manual registration and reducing the chance of forgetting to register a field.

Application Fields

// internal/config/fields.go

const GroupApplication = "Application"

func init() {
  Fields.Add(
    FieldAppEnvironment,
    FieldAppLogLevel,
    FieldAppLogOutput,
    FieldAppUpdateAuto,
    FieldAppUpdatePeriod,
  )
}

var FieldAppEnvironment = &Field{
  Name:        "environment",
  Group:       GroupApplication,
  Type:        FieldTypeString,
  Default:     "dev",
  Description: "The environment in which the application runs.",
  Example:     "prod, test, dev",
  Hidden:      true, // This field is not shown in the config list
}

var FieldAppLogLevel = &Field{
  Name:        "log.level",
  Group:       GroupApplication,
  Type:        FieldTypeString,
  Default:     "info",
  Description: "The log level to use for the application.",
  ValidValues: []any{"debug", "info", "warn", "error"},
}

var FieldAppLogOutput = &Field{
  Name:        "log.output",
  Group:       GroupApplication,
  Type:        FieldTypeString,
  Default:     nil,
  Description: "The output file to use for the application logs, if set.",
  ValidateTag: "filepath",
  Example:     "/var/log/app.log",
}

var FieldAppUpdateAuto = &Field{
  Name:        "update.auto",
  Group:       GroupApplication,
  Type:        FieldTypeBool,
  Default:     false,
  Description: "Automatically update the application when a new version is available.",
}

var FieldAppUpdatePeriod = &Field{
  Name:         "update.period",
  Group:        GroupApplication,
  Type:         FieldTypeDuration,
  Default:      "15m",
  Description:  "The period to check for updates, if enabled.",
  Docstring:    `The period can be a number of seconds, or a valid duration string.`,
  ValidateFunc: validateDuration,
  Example:      "1h, 15m, 10 (seconds)",
}

Network Fields

// fields.go

const GroupNetwork = "Network"

func init() {
  Fields.Add(
    FieldNetworkProxyHttp,
    FieldNetworkProxyHttps,
  )
}

var FieldNetworkProxyHttp = &Field{
  Name:        "proxy.http",
  Group:       GroupNetwork,
  Type:        FieldTypeString,
  Default:     nil,
  Description: "Set a proxy server for HTTP traffic",
  ValidateTag: "url",
  Example:     "http://user:password@host:port",
}

var FieldNetworkProxyHttps = &Field{
  Name:        "proxy.https",
  Group:       GroupNetwork,
  Type:        FieldTypeString,
  Default:     nil,
  Description: "Set a proxy server for HTTPS traffic",
  ValidateTag: "url",
  Example:     "http://user:password@host:port",
}

Validators

Validation is handled through the Go Validator library, which provides a declarative way to define validation rules using struct tags. The library supports a wide range of built-in validators like required, email, url, min, max, and many more. You can combine multiple validators using commas (AND logic) or pipes (OR logic). For example, validate:"required,email" ensures a field is both present and a valid email, while validate:"rgb|rgba" accepts either RGB or RGBA color formats.

In our Field structure, we use the ValidateTag field to specify these validation rules, allowing us to leverage the full power of the validator library without writing repetitive validation code.

Beyond the built-in validators, the system supports custom validation functions for complex business logic that can't be expressed through tags alone. These functions receive the value to validate and return an error if validation fails, providing complete flexibility for domain-specific rules.

// internal/config/validators.go

func validateDuration(v any) error {
  switch v := v.(type) {
  case int, int64, int32, int16, int8, uint, uint64, uint32, uint16, uint8:
    // Numeric values are valid (interpreted as seconds)
    return nil
  case float32, float64:
    // Float values are valid (interpreted as seconds)
    return nil
  case string:
    if v == "" {
      return nil // empty value is allowed
    }
    _, err := time.ParseDuration(v)
    if err != nil {
      return fmt.Errorf("invalid duration format: %s (examples: 1h30m, 15m, 10s)", v)
    }
    return nil
  default:
    return fmt.Errorf("duration must be a string or numeric value")
  }
}

Configs Powered by Viper

Viper provides the backbone for our configuration management system. It handles the complexity of merging configuration from multiple sources according to a well-defined precedence order: command-line flags override environment variables, which override config file values, which override defaults. This layered approach allows users to define base configurations in files while still being able to override specific values through environment variables in production or flags during development. Viper also manages the serialization and deserialization of configuration files in various formats, making it easy to work with YAML, JSON, TOML, or any other supported format without changing your code.

// internal/config/config.go

func Init() error {
  // Enable automatic environment variable binding
  viper.AutomaticEnv()
  viper.SetEnvPrefix(consts.ConfigEnvPrefix)
  // Replace dots with underscores in environment variable names
  viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

  // Set default values for all defined fields
  for _, field := range Fields {
    if field.Default != nil {
      viper.SetDefault(field.Name, field.Default)
    }
  }

  // Validate and process config file if specified
  cfgFile := viper.GetString(FieldFlagConfig.Name)
  if err := FieldFlagConfig.Validate(cfgFile); err != nil {
    return fmt.Errorf("config file validation failed: %w", err)
  }

  if cfgFile != "" {
    if err := loadConfigFile(cfgFile); err != nil {
      return fmt.Errorf("failed to load config file: %w", err)
    }
  }

  return nil
}

// The file extension determines the format (yaml, json, toml, etc.).
func loadConfigFile(cfgFile string) error {
  cfgFileDir := path.Dir(cfgFile)
  cfgFileBase := path.Base(cfgFile)
  cfgFileExt := path.Ext(cfgFile)
  cfgFileName := strings.TrimSuffix(cfgFileBase, cfgFileExt)

  viper.SetConfigName(cfgFileName)
  viper.SetConfigType(cfgFileExt[1:])
  viper.AddConfigPath(cfgFileDir)

  // Attempt to read the config file, but don't fail if it doesn't exist
  if err := viper.ReadInConfig(); err != nil && !errors.As(err, &viper.ConfigFileNotFoundError{}) {
    return fmt.Errorf("failed to read config file: %w", err)
  }

  return nil
}

func ReadField(f *Field) any {
  return viper.Get(f.Name)
}

func WriteField(f *Field, value any) error {
  if err := f.Validate(value); err != nil {
    return fmt.Errorf("validation failed: %w", err)
  }

  viper.Set(f.Name, value)
  return nil
}

func Save() error {
  cfgFile := viper.GetString(FieldFlagConfig.Name)
  if cfgFile == "" {
    return fmt.Errorf("no config file specified")
  }

  // Try to write the config file
  if err := viper.WriteConfigAs(cfgFile); err != nil {
    if errors.Is(err, os.ErrNotExist) {
      // Create directory structure and file if they don't exist
      return createAndWriteConfigFile(cfgFile)
    }
    return fmt.Errorf("failed to write config file %s: %w", cfgFile, err)
  }

  return nil
}

func createAndWriteConfigFile(cfgFile string) error {
  // Create the directory structure
  if err := os.MkdirAll(path.Dir(cfgFile), 0755); err != nil {
    return fmt.Errorf("failed to create config directory: %w", err)
  }

  // Create an empty config file
  if f, err := os.Create(cfgFile); err != nil {
    return fmt.Errorf("failed to create config file %s: %w", cfgFile, err)
  } else {
    f.Close()
  }

  // Write the configuration to the file
  if err := viper.WriteConfigAs(cfgFile); err != nil {
    return fmt.Errorf("failed to write configuration: %w", err)
  }

  return nil
}

Writing the CLI only once with Cobra

The beauty of our Field-Driven approach truly shines when building the CLI with Cobra. Instead of manually defining flags for each configuration parameter and keeping them in sync with validation rules and documentation, our CLI commands automatically derive everything they need from the Field definitions. This means you write the CLI structure once, and it automatically adapts as you add new configuration fields. The commands can iterate through the FieldCollection to generate flags, completions, and documentation dynamically, ensuring that the CLI always reflects the current state of your configuration schema.

The Root Command

The root command serves as the entry point for your CLI application. While you can generate the initial structure using cobra-cli init for the root command and cobra-cli add for subcommands, the real power comes from integrating it with our Field system. The root command sets up global flags and initializes the configuration system before any subcommand runs, ensuring that all parts of the application work with properly loaded and validated configuration.

// cmd/root.go

// rootCmd represents the base command when called without any subcommands.
// It serves as the entry point for the CLI application.
var rootCmd = &cobra.Command{
  Use:     consts.AppName,
  Short:   "Go application with structured configuration example",
  Long:    `A demonstration of production-ready configuration management in Go using Viper and Cobra.`,
  Version: consts.AppVersion,
}

// Execute adds all child commands to the root command and sets flags appropriately.
// This is called by main.main(). It only needs to happen once to the rootCmd.
func Execute() {
  err := rootCmd.Execute()
  if err != nil {
    os.Exit(1)
  }
}

Binding Flags

Cobra provides three types of flags: local flags (specific to a command), persistent flags (available to a command and all its subcommands), and the special PFlags type that integrates with Viper. When you bind a flag to Viper using viper.BindPFlag(), Viper automatically reads the flag value if it's set, allowing seamless integration between command-line arguments and your configuration system. This binding creates a unified interface where users can set values through flags, environment variables, or config files, and your application code doesn't need to know which source provided the value.

Our implementation uses persistent flags for global options like the config file path and verbosity level, ensuring these are available to all subcommands. The initialization happens in Cobra's OnInitialize hook, which runs before any command execution, guaranteeing that configuration is properly loaded before your command logic runs.

// cmd/root.go

var (
  FlagConfig  string // Path to the configuration file
  FlagVerbose bool   // Enable verbose output
)

func init() {
  // Initialize configuration before any command is executed
  cobra.OnInitialize(initConfig)
  // Set up persistent flags that are available to all commands
  setupPersistentFlags()
}

func setupPersistentFlags() {
  // Determine default config file location
  configDir, _ := os.UserConfigDir()
  defaultConfig := path.Join(configDir, consts.AppName, "config.yaml")

  // Config file flag
  rootCmd.PersistentFlags().StringVarP(
    &FlagConfig,
    config.FieldFlagConfig.Name,
    config.FieldFlagConfig.Shorthand,
    defaultConfig,
    config.FieldFlagConfig.Description,
  )
  viper.BindPFlag(
    config.FieldFlagConfig.Name,
    rootCmd.PersistentFlags().Lookup(config.FieldFlagConfig.Name),
  )

  // Verbose flag
  rootCmd.PersistentFlags().BoolVarP(
    &FlagVerbose,
    config.FieldFlagVerbose.Name,
    config.FieldFlagVerbose.Shorthand,
    false,
    config.FieldFlagVerbose.Description,
  )
  viper.BindPFlag(
    config.FieldFlagVerbose.Name,
    rootCmd.PersistentFlags().Lookup(config.FieldFlagVerbose.Name),
  )
}

func initConfig() {
  if err := config.Init(); err != nil {
    fmt.Printf("Configuration error: %v\n", err)
    os.Exit(1)
  }

  if FlagVerbose {
    printConfigInfo()
  }
}

func printConfigInfo() {
  cfgFile := config.ReadFieldString(config.FieldFlagConfig)
  if cfgFile != "" {
    fmt.Printf("# Using config file: %s\n", cfgFile)
  } else {
    fmt.Printf("# No config file found, using default values\n")
  }
}

The Config Commands

The configuration commands provide users with a powerful interface to interact with your application's settings. The beauty of this implementation is that these commands automatically work with any fields you define in your FieldCollection. The list command shows current values, describe provides detailed documentation, and set allows modification - all without hardcoding any specific field names. This generic approach means that adding a new configuration field automatically makes it available in all these commands without any additional code changes.

// cmd/config.go

var configCmd = &cobra.Command{
  Use:   "config",
  Short: "Configuration management commands",
  Example: `confapp config list
confapp config describe
confapp config set --log.level debug`,
}

var configListCmd = &cobra.Command{
  Use:               "list [prefix] ...",
  Short:             "List configuration values",
  RunE:              listConfig,
  ValidArgsFunction: generateFieldCompletions,
  Example: `confapp config list
confapp config list log
confapp config list proxy
confapp config list --hidden`,
}

var configSetCmd = &cobra.Command{
  Use:   "set",
  Short: "Set configuration values",
  Long:  `For more information about the configuration values, use the "describe" command.`,
  Args:  cobra.NoArgs,
  RunE:  setConfig,
  Example: `confapp config set --log.level info
confapp config set --log.level debug --log.output /var/log/app.log
confapp config set --update.auto true --update.period 1h
confapp config set --proxy.http http://proxy:8080`,
  ValidArgsFunction: generateSetCompletions,
}

var configDescribeCmd = &cobra.Command{
  Use:               "describe [prefix] ...",
  Short:             "Describe configuration parameters",
  RunE:              describeConfig,
  ValidArgsFunction: generateFieldCompletions,
  Example: `confapp config describe
confapp config describe log
confapp config describe update
confapp config describe --hidden`,
}

Generating Completions

Shell completion is one of Cobra's most powerful features, dramatically improving the user experience by providing intelligent suggestions as users type. The completion system works through ValidArgsFunction callbacks that receive the current command state and return possible completions. The args parameter contains arguments already provided, while toComplete holds the partial text being typed. The function returns a list of completion suggestions and a directive that controls shell behavior (like whether to also suggest files).

Our implementation leverages the Field definitions to provide context-aware completions. When users type config list lo, the system suggests log as a completion. When setting values with valid options, like --log.level, the completion can even suggest the valid values (debug, info, warn, error) defined in the field. This deep integration between the configuration schema and the CLI ensures users always have helpful guidance when interacting with your application.

// cmd/config.go

func generateFieldCompletions(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
  var completions []string
  for key := range config.Fields.Map() {
    if len(toComplete) == 0 || (len(key) >= len(toComplete) && key[:len(toComplete)] == toComplete) {
      completions = append(completions, key)
    }
  }
  return completions, cobra.ShellCompDirectiveNoFileComp
}

func generateSetCompletions(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
  if len(args) == 0 {
    completions := make([]string, 0, len(config.Fields))
    for _, field := range config.Fields {
      if field.Hidden {
        continue
      }
      completions = append(completions, "--"+field.Name)
    }
    return completions, cobra.ShellCompDirectiveNoFileComp
  }
  return nil, cobra.ShellCompDirectiveNoFileComp
}

func generateValidValueCompletions(validValues []any, toComplete string) ([]string, cobra.ShellCompDirective) {
  completions := make([]string, 0, len(validValues))
  for _, value := range validValues {
    if strings.HasPrefix(value.(string), toComplete) {
      completions = append(completions, value.(string))
    }
  }
  return completions, cobra.ShellCompDirectiveNoFileComp
}

Listing Configuration Values

The list command provides users with a clear view of their current configuration state. By calculating the maximum field name length, the output is neatly aligned, making it easy to scan through settings. The ability to filter by prefix allows users to focus on specific configuration groups, while the hidden flag reveals internal settings that are normally concealed. This command is essential for debugging and verifying that configuration values are being loaded correctly from all sources.

// cmd/config.go

func listConfig(cmd *cobra.Command, args []string) error {
  selectedFields := selectFieldsByPrefix(args)

  if len(selectedFields) == 0 {
    return fmt.Errorf("no configuration fields found")
  }

  // Calculate maximum field name length for alignment
  maxNameLen := calculateMaxFieldNameLength(selectedFields)

  // Display each field
  for _, field := range selectedFields {
    if field.Hidden && !FlagShowHidden {
      continue
    }
    fmt.Printf("%-*s = %v\n", maxNameLen+1, field.Name, config.ReadField(field))
  }

  return nil
}

func selectFieldsByPrefix(prefixes []string) config.FieldCollection {
  if len(prefixes) == 0 {
    return config.Fields
  }

  selectedFields := config.FieldCollection{}
  for _, prefix := range prefixes {
    for _, field := range config.Fields {
      if strings.HasPrefix(field.Name, prefix) {
        selectedFields = append(selectedFields, field)
      }
    }
  }
  return selectedFields
}

func calculateMaxFieldNameLength(fields config.FieldCollection) int {
  maxLen := 0
  for _, field := range fields {
    if len(field.Name) > maxLen {
      maxLen = len(field.Name)
    }
  }
  return maxLen
}

Setting Configuration Values

The set command demonstrates the power of our Field-Driven approach by automatically creating flags for all configuration fields. When a user runs config set --log.level debug, Cobra parses the flag, our code validates it against the Field definition, and if valid, updates the value through Viper. The command then saves the configuration to disk, ensuring changes persist across application restarts. The validation happens before any values are saved, preventing invalid configurations from being written to disk and ensuring the application always works with valid settings.

// cmd/config.go

func setConfig(cmd *cobra.Command, args []string) error {
  // Collect and process all set flags
  fieldCount := 0
  for _, field := range config.Fields {
    if cmd.Flags().Changed(field.Name) {
      if err := processFieldFlag(cmd, field); err != nil {
        return err
      }
      fieldCount++
    }
  }

  if fieldCount == 0 {
    return cmd.Help()
  }

  // Save the configuration to file
  if err := config.Save(); err != nil {
    fmt.Printf("Failed to save configuration: %v\n", err)
    os.Exit(1)
  }

  return nil
}

func processFieldFlag(cmd *cobra.Command, field *config.Field) error {
  flag := cmd.Flags().Lookup(field.Name)
  if flag == nil {
    return fmt.Errorf("flag not found: %s", field.Name)
  }

  if FlagVerbose {
    fmt.Printf("# Setting: %s: %s\n", field.Name, flag.Value.String())
  }

  if err := config.WriteField(field, flag.Value.String()); err != nil {
    return fmt.Errorf("%s: %w", field.Name, err)
  }

  return nil
}

Auto-Documentation: Describing Parameters

The describe command showcases how rich metadata in Field definitions enables automatic documentation generation. Each field's description, type, valid values, examples, and extended documentation are displayed in a structured format that helps users understand not just what a setting does, but how to use it effectively. The grouping feature organizes related fields together, making it easier to understand the relationships between different configuration options. This self-documenting nature ensures that documentation always stays in sync with the actual implementation.

// cmd/config.go

func describeConfig(cmd *cobra.Command, args []string) error {
  selectedFields := selectFieldsByPrefix(args)

  if len(selectedFields) == 0 {
    return fmt.Errorf("no configuration fields found")
  }

  // Use a writer to allow extending writing to a Pager
  w := os.Stdout

  // Display fields grouped by their group name
  for group, fields := range selectedFields.GroupIter() {
    fmt.Fprintf(w, "\n%s\n", group)

    for _, field := range fields {
      if field.Hidden && !FlagShowHidden {
        continue
      }
      writeFieldDescription(w, field)
    }
    fmt.Fprintf(w, "\n")
  }

  return nil
}

func writeFieldDescription(w interface{ Write([]byte) (int, error) }, field *config.Field) {
  // Field name and deprecation status
  if field.Deprecated != "" {
    fmt.Fprintf(w, "\n  %s (deprecated: %s)\n", field.Name, field.Deprecated)
  } else {
    fmt.Fprintf(w, "\n  %s\n", field.Name)
  }

  // Basic field information
  fmt.Fprintf(w, "    %s\n", field.Description)
  fmt.Fprintf(w, "    Type: %s\n", field.Type)

  // Current value (if different from default)
  val := config.ReadField(field)
  if val != nil && val != field.Default {
    fmt.Fprintf(w, "    Value: %v\n", val)
  }

  // Default value
  if field.Default != nil {
    fmt.Fprintf(w, "    Default: %v\n", field.Default)
  }

  // Valid values
  if field.ValidValues != nil {
    fmt.Fprintf(w, "    Valid values: %v\n", field.ValidValues)
  }

  // Validation rules
  if field.ValidateTag != "" {
    fmt.Fprintf(w, "    Validation: %s\n", field.ValidateTag)
  }

  // Example value
  if field.Example != "" {
    fmt.Fprintf(w, "    Example: %s\n", field.Example)
  }

  // Detailed documentation
  if field.Docstring != "" {
    fmt.Fprintf(w, "    Doc:\n")
    lines := strings.Split(field.Docstring, "\n")
    for _, line := range lines {
      fmt.Fprintf(w, "      %s\n", line)
    }
  }
}

Results

The Field-Driven Configuration approach delivers a powerful, user-friendly CLI that adapts automatically as your application evolves. Users benefit from intelligent completions, comprehensive documentation, and robust validation, while developers enjoy a maintainable system where adding new configuration options requires minimal code changes. The integration between Viper and Cobra through our Field abstraction creates a seamless experience where configuration can be managed through files, environment variables, or command-line flags with equal ease.

Building

The build process leverages Go's -ldflags feature to inject build-time values into the application. This allows you to customize constants like application name, version, or even default configuration values without modifying source code. This is particularly useful in CI/CD pipelines where different environments might need different defaults, or when building white-labeled versions of your application.

# Basic build
go build -o confapp

# Build with custom constants
go build -ldflags "-X github.com/lucasdecamargo/go-appconfig-example/internal/consts.AppName=myapp" -o myapp

Usage

Shell completions transform the user experience by providing context-aware suggestions as users type. Once enabled, users can press Tab to see available options, making it easy to discover configuration fields without consulting documentation. The completion system understands the command structure and provides appropriate suggestions based on context, such as showing only valid values for fields with enumerated options.

# MacOS
source <(./confapp completion zsh)
# Linux
source <(./confapp completion bash)

# List all configuration values
./confapp config list

# List specific configuration groups
./confapp config list log
./confapp config list proxy

# Describe configuration fields
./confapp config describe

# Describe specific fields
./confapp config describe log.level update

# Set configuration values
./confapp config set --log.level debug
./confapp config set --log.level info --log.output /var/log/app.log

# Show hidden fields
./confapp config list --hidden
./confapp config describe --hidden

# Verbose output
./confapp --verbose config list

Checking the documentation with describe

Retrieving values with list

Generating completions for the list command

Generating completions for the flags in set command

Generating completions for the flag values in set command

Conclusion

The Field-Driven Configuration pattern presented in this article demonstrates how thoughtful abstraction can transform configuration management from a maintenance burden into a self-maintaining system. By defining configuration fields as first-class entities with rich metadata, we've created a solution that respects the Open-Closed Principle while providing exceptional developer and user experiences.

The integration of Viper, Cobra, and Go Validator through our Field abstraction eliminates the common pain points of configuration management: keeping documentation in sync, maintaining validation rules, and providing good CLI experiences. The result is a system where adding new configuration options is as simple as defining a new Field struct, and everything else - from CLI flags to validation to documentation - automatically adapts.

This approach scales elegantly from simple applications with a handful of settings to complex systems with hundreds of configuration parameters organized into logical groups. The automatic generation of completions and documentation ensures that as your application grows, it remains discoverable and user-friendly.

Extending the Solution

The architecture presented here provides a solid foundation that can be extended in several ways:

Configuration Profiles: Add support for multiple configuration profiles (development, staging, production) by extending the FieldCollection to support profile-specific overrides while maintaining the same validation and documentation capabilities.

Dynamic Reloading: Implement configuration hot-reloading using Viper's WatchConfig functionality, with the Field definitions providing the validation layer to ensure changes are safe before applying them.

API Integration: Generate OpenAPI specifications or GraphQL schemas from Field definitions, ensuring that API documentation stays synchronized with configuration capabilities.

Nested Structures: Extend the Field structure to support complex nested configurations while maintaining the same validation and documentation benefits.

For a complete implementation with additional features like configuration profiles, pager support for long documentation, and more sophisticated validation examples, check out my GitHub repository.

lucasdecamargo / go-appconfig-example

Guidelines for setting up configuration parameters in large Go applications with Viper and Cobra.

Go Application Configuration Template

A production-ready template for managing configuration parameters in Go applications using Viper and Cobra. This template demonstrates a clean, maintainable approach to configuration management with a single source of truth for all configuration metadata.

🎯 Key Features

Single Source of Truth: Configuration fields are defined once with all metadata (validation, documentation, defaults)
Type Safety: Strongly typed configuration with validation
CLI Integration: Seamless integration with Cobra for command-line interfaces
Multiple Formats: Support for YAML, JSON, TOML, HCL, and ENV files
Environment Variables: Automatic binding with configurable prefix
Shell Completion: Auto-completion for configuration field names and values
Documentation: Built-in help and documentation generation
Validation: Multiple validation strategies (tags, custom functions, valid values)

🏗️ Architecture Overview

Core Concept: Field-Driven Configuration

The central idea is to define configuration fields as structured data that contains everything needed to:

Validate values
Generate CLI…

View on GitHub

The repository includes comprehensive examples and can serve as a starting template for your own production applications. Feel free to star the repository if you find it useful, and don't hesitate to open issues or contribute improvements!

DEV Community: Lucas de Camargo

AWS Governance with a Terragrunt Live Environment

Getting Started

lucasdecamargo / terragrunt-live-example

Template for a Terragrunt stack using my AWS Terraform modules including Governance management.

terragrunt-live-example

Installation

Named Profiles

Live Environment

Writing the Terragrunt Stack

Directory Structure

Locals

The Root File

Generating Providers

Remote State Files

Units (Resources)

Deploying with Terragrunt

Deploying the Remote State Backend

Deployment Scopes

What's Next?

Configuring AWS Named Profiles

Single Sign-On (SSO) - Recommended

Access Keys

Testing Profiles

Deploying Governance before SSO

What's Next?

Introducing the Three AWS Governance Pillars with Terraform

The Three Governance Pillars

Terraform Modules

lucasdecamargo / terraform-aws-governance

Terraform modules for AWS governance services, like Organizations, Identity and Budget.

Terraform AWS Governance Modules

Module Overview

organization

Organization

Organization Unit

Account

What's Next?

Go Beyond Viper and Cobra: Declarative Field-Driven Configuration for Go Apps

Table of Contents

Requirements

Functionalities

The App Example

Application Fields

Network Fields

Implementation

File Structure

Constants

Field-Driven Configuration

The Field Structure

Field Collection

Real Field Definitions

Application Fields

Network Fields

Validators

Configs Powered by Viper

Writing the CLI only once with Cobra

The Root Command

Binding Flags

The Config Commands

Generating Completions

Listing Configuration Values

Setting Configuration Values

Auto-Documentation: Describing Parameters

Results

Building

Usage

Conclusion

Extending the Solution

lucasdecamargo / go-appconfig-example

Guidelines for setting up configuration parameters in large Go applications with Viper and Cobra.

Go Application Configuration Template

🎯 Key Features

🏗️ Architecture Overview

Core Concept: Field-Driven Configuration

Buy me a coffee