DEV Community: Cybergavin

Introducing s3-compliance: An OpenTofu Module for Secure and Standardized S3 Buckets

Cybergavin — Sat, 18 Jan 2025 06:20:33 +0000

In this blog post, I give you an overview of the s3-compliance OpenTofu module, for provisioning and managing Amazon S3 buckets, while adhering to data classification standards.

What is s3-compliance?

S3-compliance is an opinionated OpenTofu module designed to:

Create and manage one or more S3 buckets using a parameterized deployment.
Embed organizational standards and policies into your infrastructure provisioning, based on data classifications.
Provide flexibility for users to specify certain settings while enforcing pre-approved defaults.

Not all types of data require every S3 bucket feature such as versioning or object lock. The s3-compliance module addresses these diverse needs by allowing different sets of configurations for different data classifications. This tailored approach ensures that buckets are configured with only the necessary features for their specific data classification, striking a balance between security, efficiency and cost. With this module, teams can focus on their work while ensuring their provisioned S3 buckets align with organization standards and recommended practices.

Data Classifications

The s3-compliance module uses data classifications (e.g., public, internal, compliance) to determine which features to mandate. For example:

Public Data: Minimal security features, with public access intentionally enabled where appropriate.
Internal Data: Encryption using KMS and logging are mandatory; versioning is optional.
Regulated Data: Strict compliance requirements, including encryption, versioning, object lock, and logging.

By matching user-provided configurations to data classifications, the module allows flexibility for some settings, while enforcing others, thereby ensuring efficiency and compliance.

Compliance File

The s3-compliance.tf file embedded in the module defines organizational standards for various data classifications, including:

Public access restrictions
Encryption settings
CloudTrail Logging
Versioning
Object locks for immutable storage

Security teams vet this file to ensure compliance with organizational and regulatory requirements. A workflow may be triggered upon any changes to this compliance file to solicit the security team's approval. Alternatively, the security team may host this file in their own repository and make it available as a module output for consumption by the s3-compliance module.

Input parameters

Given below is the variable structure for the input parameters, showing that the s3_buckets variable accepts a list of S3 buckets for provisioning. Most of the optional configurations have defaults pre-defined as per the s3-compliance file.

variable "s3_buckets" {
  description = "List of S3 bucket configurations."
  type = list(object({
    # Basic configuration
    name                  = string
    data_classification   = string # Defaults are "public", "internal" and "compliance"
    public_access_enabled = optional(bool)
    versioning_enabled    = optional(bool)
    logging_enabled       = optional(bool)
    tags                  = optional(map(string), {})

    # Encryption settings
    kms_master_key_id   = optional(string) # Use S3-managed keys by default
    compliance_standard = optional(string) # e.g., "PCI-DSS", "HIPAA", "ISO27001"

    # Object Lock settings
    object_lock = optional(object({
      mode           = optional(string) # "GOVERNANCE" or "COMPLIANCE"
      retention_days = optional(number) # Number of days to retain objects in locked state
    }), null)

    # Lifecycle configuration
    lifecycle_transitions = optional(object({
      intelligent_tiering_days = optional(number, null) # Days for transitioning objects to the Intelligent Tiering class
      glacier_ir_days          = optional(number, null) # Days for transitioning objects to the Glacier Instant Retrieval class
      glacier_fr_days          = optional(number, null) # Days for transitioning objects to the Glacier Flexible Retrieval class
      glacier_da_days          = optional(number, null) # Days for transitioning objects to the Glacier Deep Archive class
    }), null)

    expiration_days = optional(number, null) # Expiration after the latest transition
  }))

Example Usage

The s3-compliance module may be used as shown below.

# Provision S3 buckets for a sales application
module "sales-s3" {
  source = "git::https://github.com/cybergavin/terraform-aws-s3-compliance.git?ref=57e686755fd9c0c89b57ba1babe3f741ad6d6515
  org         = var.org
  app_id      = var.app_id
  environment = var.environment
  s3_buckets  = var.s3_buckets
  s3_logs     = var.s3_logs
  global_tags = var.global_tags
}

An example s3_buckets variable for provisioning three S3 buckets for a sales application, is given below.

s3_buckets = [
    {
        name = "catalogs"
        data_classification = "public"
        public_access_enabled = true
        tags = {
            "description" = "Sales product catalogs"
        }
    },
    {
        name = "inventory"
        data_classification = "internal"
        tags = {
            "description" = "Sales inventory"
        }
        lifecycle_transitions = {
            intelligent_tiering_days = 180
        }
        expiration_days = 365
    },
    {
        name = "payment"
        data_classification = "compliance"
        compliance_standard = "PCI-DSS"
        tags = {
            "description" = "Sales payment transactions"
        }
        object_lock = {
            mode = "COMPLIANCE"
            retention_days = "2555"
        }
        lifecycle_transitions = {
            intelligent_tiering_days = 180
            glacier_ir_days = 365
            glacier_fr_days = 730
        }
        expiration_days = 2555
    }
]

For a more detailed example of using the module refer the sales-s3 example.

Getting Started

Clone the module repository from GitHub: git clone https://github.com/cybergavin/terraform-aws-s3-compliance.git
Review the s3-compliance.tf file with your security team and configure it to meet your data classification standards.
Integrate the module into your OpenTofu project.
Customize configurations where needed, while adhering to enforced defaults and use the module as per the example provided.

CI/CD workflow for module build

The module repository includes a GitHub Actions CI/CD workflow used for building the module that includes linting (tofu fmt and tflint), scanning (checkov) , testing in a sandbox environment, creating a pre-release tag, creating a pull request and create a stable release tag. This built-in CI/CD workflow could be a good starting point for you to maintain the module, if required. Alternatively, you may integrate the module into your ecosystem (TACOS, Atlantis, Digger, HCP Cloud, etc.) and use the available features for a build and release workflow.

Conclusion

The s3-compliance OpenTofu module bridges the gap between flexibility and security, allowing teams to provision S3 buckets that meet organizational standards easily. By embedding compliance directly into the provisioning process, this module simplifies operations, reduces risks, and accelerates infrastructure deployment.

Check out the s3-compliance module on GitHub and get started today! Feel free to leave me feedback on this blog post and/or raise issues on GitHub.

OpenTofu module for spoke VPC with TGW

Cybergavin — Sun, 05 Jan 2025 06:46:44 +0000

In this blog post, I’ll describe the vpc-spoke-tgw OpenTofu module that simplifies the process of setting up networking for workload/spoke VPCs attached to a Transit Gateway (TGW). This module automates the creation of essential networking resources, including VPCs, subnets, route tables, security groups, DHCP option sets, and TGW attachments, facilitating integration with your organization’s hub-and-spoke architecture. This module also leverages the terraform-null-label module to provide consistent naming for provisioned resources.

OpenTofu modules can be built and delivered as products, allowing other IT teams to use them independently.

An OpenTofu module for Hub-Spoke Deployments

The image below represents a typical hub-spoke architecture with the following:

Hub: An AWS account containing core infrastructure services (network ingress/egress, TGW, security appliances, shared services).

Spoke: An AWS account hosting a business application or workload with ingress/egress traffic from/to the Internet via the TGW

The hub shares its TGW with the spoke via AWS Resource Access Manager (RAM).

The vpc-spoke-tgw OpenTofu module accepts the TGW share invitation and sets up basic networking in the spoke (VPC, subnets, DHCP options, route tables with routes to the TGW, security groups).

Use Case: Setting Up Secure, Standardized Networking for a Multi-Account AWS Environment

Scenario:

Your organization has a multi-account AWS environment where the network team manages the central AWS account (hub), which includes the Transit Gateway (TGW), firewall configurations, and security measures like ingress/egress traffic management. The application teams manage their own AWS accounts, where they deploy various workloads (e.g., web applications, APIs, etc.) and need secure connectivity to other services within the company’s AWS and on-premises networks as well as to the Internet. In some organizations, infrastructure or platform teams may set up workload AWS accounts before application teams deploy their workloads.

To enable this, the application/infrastructure/platform teams attach their VPCs to the TGW shared by the network team. However, each team requires a standardized, secure, and automated way to create their VPCs and configure networking components like subnets, route tables, security groups, and DHCP options.

Solution:

The vpc-spoke-tgw OpenTofu module is designed to automate the networking setup in these workload VPCs that need to connect to the shared TGW. The module streamlines and standardizes the process, ensuring consistent and secure networking for each team.

Key Features of the `vpc-spoke-tgw` OpenTofu Module:

VPC & Subnet Creation: Creates a VPC with private subnets distributed across availability zones, when multiple subnets are configured.
TGW Attachment: Accepts the TGW share and creates the attachment of the newly created VPC to the Transit Gateway managed by the network team, ensuring proper routing and connectivity.
Route Table Setup: Creates route tables with routes to the TGW for secure communication between the application’s VPC and other VPCs or on-premises systems via the Transit Gateway.
Security Groups: Creates security groups to control inbound and outbound traffic based on application-specific needs.
DHCP Option Set: Creates a DHCP option set for proper domain name resolution across the VPC.
Parameterized Deployment: Can be used across multiple environments by using different input parameters or variables, specified in .tfvars files, thereby facilitating reusability, consistency and automation (CI/CD pipelines).
Standardized Deployment: Leverages the terraform-null-label module to provide consistent names for provisioned resources and allows for global tags, thereby facilitating adherence to naming and tagging standards.

Example Workflow:

Network Team: Shares the TGW with application teams via AWS Resource Access Manager (RAM), sets the necessary security configurations and provides CIDR blocks (reserved in IPAM) for use by application teams for VPCs and subnets.
Application Teams: Use the vpc-spoke-tgw OpenTofu module to create their VPC and related networking resources (subnets, route table, security groups, etc.) with a simple configuration. The application teams can then deploy their applications into their networks an easily scale their networking (e.g., additional subnets, security groups) autonomously.

Benefits:

Consistency: All application teams follow the same networking standards, simplifying management.
Security: By automatically setting up secure VPCs, subnets, and security groups, the module ensures that networking is configured in accordance with your organization's security policies.
Efficiency: Application teams can focus on their workloads, while the OpenTofu module handles the repetitive networking setup tasks.
Scalability: As the company grows and more application teams are onboarded, the module allows for easy and automated scaling of networking infrastructure without additional manual effort.

How to access the `vpc-spoke-tgw` OpenTofu module

Access the module from my GitHub repo:

GitHub - The module's repo contains GitHub Actions workflows for CI/CD pipelines used for module testing as well as an example on using the module.

Conclusion

In this blog post, I demonstrated how you may use the vpc-spoke-tgw OpenTofu module to streamline the setup of networking for application teams while ensuring consistency, security, and compliance with organization policies. Try it out and feel free to provide feedback either as a comment on this blog post or as an issue in the GitHub repo. This is an example of how Platform Engineering teams may deliver IaC products for easy and autonomous consumption by application teams. There are many ways to deliver IaC products. You may provide access to the module via a Git repo, a UI, use TACOS software, etc., but you'll need to pick an approach that works well for your environment (infrastructure, policies, practices, culture).

DEV Community: Cybergavin

Introducing s3-compliance: An OpenTofu Module for Secure and Standardized S3 Buckets

What is s3-compliance?

Data Classifications

Compliance File

Input parameters

Example Usage

Getting Started

CI/CD workflow for module build

Conclusion

OpenTofu module for spoke VPC with TGW

An OpenTofu module for Hub-Spoke Deployments

Use Case: Setting Up Secure, Standardized Networking for a Multi-Account AWS Environment

Scenario:

Solution:

Key Features of the vpc-spoke-tgw OpenTofu Module:

Example Workflow:

Benefits:

How to access the vpc-spoke-tgw OpenTofu module

Conclusion

Key Features of the `vpc-spoke-tgw` OpenTofu Module:

How to access the `vpc-spoke-tgw` OpenTofu module