Study Notes 1.3.2: Terraform Basics with GCP

#dataengineering #dezoomcamp #terraform #beginners

1. Authentication Setup for Terraform

Service Account Creation:
- A service account is used by software (like Terraform) to authenticate with GCP.
- Steps:
  1. Navigate to IAM & Admin > Service Accounts in GCP Console.
  2. Create a service account (e.g., terraform-runner).
  3. Assign permissions:
    - Storage Admin (for GCS bucket management).
    - BigQuery Admin (for dataset management).
    - Compute Engine Admin (optional, added later via "Edit Principal").
Service Account Key:
- Generate a JSON key for the service account (Manage Keys > Create New Key > JSON).
- Security Warning:
  - Never expose the JSON key (risk of unauthorized resource creation, cost overruns, or malicious activity).
  - Avoid storing keys in insecure locations (email, Google Drive, GitHub).

2. Local Environment Configuration

Directory Setup:
- Create a project directory (e.g., terraform-demo) and a subdirectory for keys (e.g., keys/).
- Save the JSON key as keys/my-creds.json.

Environment Variable:

Set the key path for Terraform authentication:

export GOOGLE_APPLICATION_CREDENTIALS=~/terraform-demo/keys/my-creds.json

VS Code Setup:
- Install the HashiCorp Terraform extension for syntax highlighting and autocompletion.

3. Terraform Configuration

Provider Setup:

Create main.tf with the Google provider configuration:

provider "google" {
  project = "your-project-id"  # Use GCP project ID, not name
  region  = "us-central1"
}

Formatting: Use terraform fmt to auto-format code.

4. Resource Creation (GCS Bucket Example)

Define a Bucket:

resource "google_storage_bucket" "demo-bucket" {
  name          = "terraform-demo-bucket"  # Globally unique name
  location      = "US"
  force_destroy = true  # Allows Terraform to delete non-empty buckets

  lifecycle_rule {
    action {
      type = "Delete"
    }
    condition {
      age = 3  # Delete objects after 3 days
    }
  }

  lifecycle_rule {
    action {
      type = "AbortIncompleteMultipartUpload"
    }
    condition {
      age = 1  # Abort incomplete uploads after 1 day
    }
  }
}

Workflow Commands:
1. terraform init: Initializes providers and modules.
2. terraform plan: Previews changes without applying them.
3. terraform apply: Creates resources (type yes to confirm).
4. terraform destroy: Removes all managed resources (type yes to confirm).

5. Security Best Practices

State File (terraform.tfstate):
- Contains sensitive data (resource IDs, configurations).
- Never commit to version control. Use .gitignore (see below).

GitHub Precautions:

Add a .gitignore file to exclude:

# .gitignore
*.tfstate
*.tfstate.backup
*.json  # Exclude credential files
.terraform/

Use private repositories for Terraform projects.

6. Key Takeaways

Least Privilege: Assign minimal permissions to service accounts.
Credentials Management:
- Rotate keys regularly.
- Use environment variables or secure secret managers.
State Management:
- Store terraform.tfstate securely (e.g., GCS bucket with versioning).
Plan Before Apply: Always review terraform plan to avoid unintended changes.