Solved: Am I the only one who builds in the Console first, then reverse engineers the IaC?

🚀 Executive Summary

TL;DR: Many DevOps engineers face configuration drift and ‘snowflake’ environments by building infrastructure manually in cloud consoles before translating it to IaC. The solution involves adopting an IaC-first approach with dedicated sandbox environments, leveraging reverse engineering tools for existing resources, and implementing policy enforcement and drift detection to maintain IaC as the single source of truth.

🎯 Key Takeaways

• Adopt an ‘IaC-First with Sandbox’ strategy, providing engineers with isolated, disposable cloud accounts to write and deploy IaC (e.g., Terraform, CloudFormation) from the outset, using the console primarily for observation.
• Utilize cloud provider import and reverse engineering tools like Terraform Import, Terraformer, and Azure Export Template to bring existing console-built resources under IaC management, generating initial configurations.
• Implement incremental refactoring, automated drift detection (e.g., terraform plan, AWS CloudFormation Drift Detection), and Policy-as-Code (e.g., AWS SCPs, OPA) to enforce IaC best practices and prevent manual configuration changes in higher environments.

Many DevOps engineers find themselves building infrastructure manually in cloud consoles before translating it to IaC. This post explores why this “console-first” approach happens and provides practical strategies to shift towards a more efficient, IaC-driven workflow without sacrificing agility.

Understanding the “Console-First” Symptom

It’s a familiar scenario: you need a new S3 bucket, a quick EC2 instance for testing, or a Function App to prototype a new feature. You log into the AWS, Azure, or GCP console, click through the wizards, and within minutes, your resource is live. It’s fast, intuitive, and offers immediate gratification. This “console-first” approach is particularly common when:

• Learning New Services: Exploring unfamiliar cloud services often starts with the console to grasp concepts visually.
• Rapid Prototyping: For quick proofs-of-concept or throwaway environments, the console feels like the fastest path.
• Troubleshooting/Hotfixes: Under pressure, directly tweaking a setting in the console can seem quicker than going through an IaC pipeline.
• Lack of IaC Maturity: Teams new to IaC or those without established pipelines may default to manual creation.

However, this expediency comes at a significant cost, leading to:

• Configuration Drift: The production environment diverges from the source of truth (your IaC repository), making deployments unpredictable.
• Lack of Version Control: Manual changes are untracked, unreviewable, and difficult to roll back.
• “Snowflake” Environments: No two environments are identical, hindering consistency and reproducibility.
• Security and Compliance Risks: Manual console access increases the attack surface and makes auditing challenging.
• Scalability and Efficiency Bottlenecks: Manual processes don’t scale; automating repeat tasks is impossible without IaC.

The Reddit thread title “Am I the only one who builds in the Console first, then reverse engineers the IaC?” highlights that this is a widespread challenge, not an isolated incident. Let’s explore practical solutions to address it.

Solution 1: Adopt a “IaC-First with Sandbox” Strategy

The most effective long-term solution is to shift your mindset and processes to be IaC-first. However, recognizing the need for experimentation, this strategy integrates dedicated sandbox environments.

How it Works:

• Dedicated Sandbox Accounts/Projects: Provide each engineer or team with their own, isolated cloud account (e.g., AWS account, Azure subscription, GCP project). These are low-cost, disposable environments.
• IaC as the Primary Method: Encourage engineers to write Terraform, CloudFormation, ARM templates, or Pulumi code directly from the outset, even for experimental work.
• Console for Observation (and emergency tweaks): The console becomes a tool for observing the state of resources deployed by IaC, reviewing logs, and confirming configurations. If a quick manual tweak is made in the sandbox, the expectation is to immediately update the IaC and apply it.
• Automated Teardown: Implement policies or automated scripts to tear down sandbox resources after a set period to control costs and prevent accumulation of untracked resources.

Real Examples and Configuration:

Imagine an engineer needs to test a new Lambda function triggered by an S3 event. Instead of creating it directly in the console:

1. They write the Terraform for the S3 bucket, the Lambda function, and the necessary IAM roles in their local environment.
2. They deploy it to their personal sandbox account using terraform apply.
3. They can then use the console to verify the deployment, check logs, and test the function. If a minor change is needed (e.g., a specific environment variable for a quick test), they make it in the console, immediately update their Terraform, and run terraform apply again to reconcile.

Here’s a basic Terraform example for an S3 bucket in a sandbox:

# main.tf
resource "aws_s3_bucket" "my_sandbox_bucket" {
  bucket = "my-unique-sandbox-bucket-${random_id.suffix.hex}"
  acl    = "private"

  tags = {
    Owner       = "engineer-name"
    Environment = "sandbox"
    Purpose     = "experimentation"
  }
}

resource "random_id" "suffix" {
  byte_length = 8
}

output "bucket_name" {
  value = aws_s3_bucket.my_sandbox_bucket.id
}

To enforce cost control and cleanup, you might use AWS Organizations Service Control Policies (SCPs) to limit resource types or regions in sandbox accounts, or implement scheduled Lambda functions to delete resources tagged with “sandbox” older than X days.

# Example AWS SCP (JSON) - Prevents creation of large EC2 instances in sandbox
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Action": [
                "ec2:RunInstances"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:instance/*"
            ],
            "Condition": {
                "ForAnyValue:StringLike": {
                    "ec2:InstanceType": [
                        "*.large",
                        "*.xlarge",
                        "*.2xlarge",
                        "*.4xlarge",
                        "*.8xlarge",
                        "*.12xlarge",
                        "*.16xlarge",
                        "*.24xlarge"
                    ]
                }
            }
        }
    ]
}

Solution 2: Leverage Cloud Provider Import/Reverse Engineering Tools

When you *do* find yourself with existing console-built resources that need to be brought under IaC management, or for generating initial IaC from an existing setup, specialized tools can help.

How it Works:

• Terraform Import: Terraform has a built-in import command to bring existing resources into its state file. This allows you to manage previously manually created resources with Terraform without recreating them.
• Terraformer (Google/Palo Alto Networks): A powerful open-source tool that can generate Terraform HCL configuration from existing resources across multiple cloud providers (AWS, Azure, GCP, Kubernetes). It’s excellent for “reverse engineering” an entire environment.
• Azure Export Template: The Azure portal provides a direct option to “Export template” for individual resources or entire resource groups, generating an ARM template.

Real Examples and Configuration:

Terraform Import:

Let’s say you manually created an S3 bucket named my-important-console-bucket. To bring it under Terraform management:

1. Define the resource in your Terraform configuration (e.g., main.tf) as if you were going to create it.
2. Run the terraform import command.

# In your main.tf (before import)
resource "aws_s3_bucket" "existing_bucket" {
  bucket = "my-important-console-bucket"
  # You might need to add other attributes like acl, region, tags here
  # to match the existing bucket's configuration after import.
}

terraform import aws_s3_bucket.existing_bucket my-important-console-bucket

After import, run terraform plan to see if there’s any drift and adjust your HCL to match the existing state accurately. This often requires careful manual review and refinement of the generated configuration.

Terraformer (for AWS):

To generate Terraform HCL for all S3 buckets in a specific AWS region:

terraformer import aws --resources=s3 --regions=us-east-1

This command will generate .tf files and a .tfstate file in a new directory, representing your existing S3 buckets in us-east-1. You then have a starting point for managing these resources with Terraform.

Azure Export Template:

In the Azure Portal, navigate to a resource group or a specific resource (e.g., a Virtual Network).

On the left-hand menu, scroll down to “Automation” and click “Export template.”

You’ll see a JSON ARM template that defines the resource(s). You can download it or copy it for use in your IaC pipelines.

# Example of a simplified ARM template snippet you might export
{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "resources": [
    {
      "type": "Microsoft.Storage/storageAccounts",
      "apiVersion": "2021-09-01",
      "name": "mystorageaccount001",
      "location": "eastus",
      "sku": {
        "name": "Standard_LRS"
      },
      "kind": "StorageV2",
      "properties": {}
    }
  ]
}

Comparison of Reverse Engineering Tools/Approaches:

Feature	Terraform Import	Terraformer	Azure Export Template
Purpose	Bring existing resources under TF state management.	Generate full HCL from existing infra.	Export ARM template for Azure resources.
Scope	One resource at a time (though can be scripted).	Multiple resources, entire environments.	Individual resource or resource group.
Output	Updates Terraform state, requires HCL definition.	Full HCL files (.tf) and .tfstate.	JSON ARM template.
Cloud Support	Terraform-supported providers (many).	AWS, Azure, GCP, Kubernetes, etc.	Azure only.
Complexity	Medium, requires manual HCL matching.	Low for basic generation, higher for cleanup.	Low, direct from portal.
Ideal Use Case	Gradually onboarding existing resources to IaC.	Bootstrapping IaC for an existing environment.	Capturing the current state of Azure resources for review or redeployment.

Solution 3: Incremental Refactoring and Policy Enforcement

Shifting from a console-first to an IaC-first culture is a journey. This solution focuses on a gradual transition combined with guardrails to prevent regression.

How it Works:

• Identify Critical/High-Drift Components: Start by bringing the most critical, frequently changed, or production-sensitive resources under IaC. Prioritize areas known for drift.
• Phased Console Access Restrictions: Gradually restrict direct console write access for non-emergency operations in higher environments (staging, production). Rely on CI/CD pipelines for deployments.
• Automated Drift Detection & Auditing: Implement tools to continuously monitor your cloud environments for deviations from your IaC.
• Policy-as-Code (PaC): Define policies that enforce IaC best practices, security standards, and cost controls.
• Training and Cultural Shift: Educate teams on the “why” behind IaC, provide training on tools, and foster a culture of automation and collaboration.

Real Examples and Configuration:

Automated Drift Detection:

AWS CloudFormation Drift Detection: For CloudFormation stacks, you can use the AWS CLI to detect drift:

aws cloudformation detect-stack-drift --stack-name MyProductionAppStack
aws cloudformation describe-stack-resource-drifts --stack-name MyProductionAppStack

Terraform plan: Running terraform plan against an already deployed environment will show differences between your local IaC and the remote state, effectively acting as a drift detector.

terraform plan

This should be part of your CI/CD pipeline, perhaps on a schedule, to alert on unexpected changes.

Policy-as-Code (PaC):

AWS Service Control Policies (SCPs): These are part of AWS Organizations and can enforce maximum permissions for member accounts. For instance, you could deny console access for IAM users in production accounts, forcing API-only or role-based access which can be managed by IaC.

# Example AWS SCP to deny console access for specific actions
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Action": [
                "s3:PutObject",
                "ec2:RunInstances"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:CalledVia": "false"
                }
            }
        }
    ]
}

This SCP denies direct console actions for s3:PutObject and ec2:RunInstances, meaning these operations must originate from an API call (e.g., via a CLI, SDK, or IaC tool).

Open Policy Agent (OPA) / Gatekeeper (Kubernetes): For more granular, context-aware policies, OPA allows you to define policies in Rego language that can be applied across your infrastructure, including Kubernetes, API gateways, CI/CD pipelines, and even Terraform plans (via tools like conftest).

# Example Rego policy for OPA (e.g., check Terraform plan for unencrypted S3 buckets)
package terraform.aws.s3

deny[msg] {
    some i
    resource := input.resource_changes[i]
    resource.type == "aws_s3_bucket"
    resource.change.after.bucket_prefix == null # Catching new buckets
    not resource.change.after.server_side_encryption_configuration.rule[0].apply_server_side_encryption_by_default.sse_algorithm == "AES256"
    msg := sprintf("S3 bucket %s must enforce server-side encryption.", [resource.address])
}

This policy would be integrated into your CI/CD pipeline to reject Terraform plans that attempt to create unencrypted S3 buckets, preventing manual overrides or accidental insecure configurations.

Conclusion

While the cloud console offers undeniable convenience for quick experimentation and learning, relying on it for managing production infrastructure introduces significant risks and technical debt. By adopting an IaC-first mindset, leveraging sandbox environments, utilizing reverse engineering tools where necessary, and incrementally enforcing policies, DevOps teams can successfully transition away from the “console-first, reverse engineer later” trap. The goal is not to abandon the console entirely, but to relegate it to its appropriate role: an observation and occasional troubleshooting tool, with IaC as the single source of truth for your infrastructure.

---

👉 Read the original article on TechResolve.blog