DEV Community

Tom
Tom

Posted on Originally published at mulgadc.com

Using Terraform to Manage On-Premise Infrastructure

#devops #terraform #aws #opensource

Most Terraform configs that target AWS are more portable than their authors expect. The resource definitions, modules, variable files, and state are all completely decoupled from which endpoint the provider talks to. When you point the AWS provider at Spinifex, a fully AWS-compatible cloud platform running on your own hardware, the resources you've already written apply without modification. What changes is a single block in your provider configuration.

This post covers the Terraform-specific details: the two-endpoint architecture, the provider flags that AWS requires but that Spinifex doesn't enforce, state management, and the workbooks in the Spinifex repository that cover common infrastructure patterns.

The two-endpoint architecture

Spinifex exposes two endpoints. The main AWS gateway runs on port 9999 and handles EC2, IAM, STS, EKS, ECR, ECS, VPC, ELB, and other compute and identity services. S3-compatible object storage is served by Predastore on port 8443. The AWS provider routes different services to different endpoints, and S3 requires an additional configuration flag.

A provider block targeting compute services only looks like this:

variable "spinifex_endpoint" {
  type    = string
  default = "https://127.0.0.1:9999"
}

provider "aws" {
  region = "ap-southeast-2"

  endpoints {
    ec2 = var.spinifex_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}
Enter fullscreen mode Exit fullscreen mode

When your config also provisions S3 resources, add the Predastore endpoint and enable path-style addressing:

variable "predastore_endpoint" {
  type    = string
  default = "https://127.0.0.1:8443"
}

provider "aws" {
  region = "ap-southeast-2"

  endpoints {
    ec2 = var.spinifex_endpoint
    s3  = var.predastore_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
  }

  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}
Enter fullscreen mode Exit fullscreen mode

The s3_use_path_style flag is required because Predastore uses path-style bucket addressing (https://host:8443/bucket-name) rather than virtual-hosted-style (https://bucket-name.host). Without it, the provider will generate request URLs that Predastore does not recognize.

The skip_* flags

AWS imposes several validation checks before an API call reaches the provider. These checks are designed to catch misconfiguration when targeting real AWS, but they test against AWS-specific infrastructure (the STS GetCallerIdentity endpoint, the AWS metadata API) that doesn't apply to Spinifex. The four skip_* flags disable those checks:

  • skip_credentials_validation: skips the initial STS check that validates the access key against AWS
  • skip_metadata_api_check: skips the check for the EC2 instance metadata service, which is only relevant when running inside AWS
  • skip_requesting_account_id: skips the sts:GetCallerIdentity call used to populate the account ID in resource ARNs
  • skip_region_validation: skips the check that the configured region is a known AWS region

All four are required. Omitting any one of them causes the provider initialization to fail before it sends a single resource request to Spinifex.

Passing endpoint addresses

The defaults in the variable declarations (https://127.0.0.1:9999 and https://127.0.0.1:8443) are useful for running Terraform locally on the same machine as Spinifex but won't work when Terraform runs on a separate host. Override them with a terraform.tfvars file:

spinifex_endpoint   = "https://192.168.1.10:9999"
predastore_endpoint = "https://192.168.1.10:8443"
Enter fullscreen mode Exit fullscreen mode

Or via environment variables, which avoids committing host addresses to source control:

export TF_VAR_spinifex_endpoint="https://192.168.1.10:9999"
export TF_VAR_predastore_endpoint="https://192.168.1.10:8443"
Enter fullscreen mode Exit fullscreen mode

The environment variable form works well in CI pipelines where the Spinifex host address is an infrastructure secret that shouldn't appear in repository files.

AMI data sources

Terraform configs that look up AMIs with data "aws_ami" need one adjustment. AWS AMIs are owned by specific AWS account IDs, but Spinifex uses 000000000000 as the placeholder account for imported images. The filter block looks like this:

data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["000000000000"]

  filter {
    name   = "name"
    values = ["*ubuntu-24.04*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}
Enter fullscreen mode Exit fullscreen mode

Configs that filter by a real AWS account ID (such as 099720109477 for Canonical's official Ubuntu AMIs) won't match any Spinifex images. Changing owners to ["000000000000"] is the only modification required.

State

Local state works with Spinifex exactly as it does with AWS. The documented workbooks all use a local state backend:

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Remote state using Predastore as an S3 backend is not covered in the documented workbooks. If you need shared remote state for a team workflow, use a local backend with a shared filesystem or an external state store such as Terraform Cloud. Plan for this before you're managing non-trivial infrastructure with multiple contributors.

Credentials

Terraform picks up credentials from the same places as the AWS CLI: environment variables, the ~/.aws/credentials file, or an instance profile if Terraform is running on a Spinifex-launched instance. After spx admin init, the profile named spinifex-<nodename> has the access key and secret for the node. Set AWS_PROFILE=spinifex-<nodename> in your environment and the provider will use it without any credential configuration in the HCL.

The workbooks

The Spinifex repository includes a set of Terraform workbooks that provide complete, working configurations for common infrastructure patterns. They're organized under docs/terraform-workbooks and cover:

  • bastion-private-subnet: a VPC with public and private subnets, a bastion host, and a private instance behind it
  • s3-webapp: a Flask application on EC2 that reads from Predastore using IMDS credentials via an instance profile, with no long-lived S3 keys baked into the instance
  • nginx-webserver: a simple EC2-hosted web server behind an Application Load Balancer
  • nginx-alb: a minimal ALB configuration for routing traffic to EC2 instances
  • rds-quickstart: a VPC, subnet group, and RDS instance connected to an EC2 app server
  • ecs-quickstart: a full ECS stack including VPC, IAM, cluster, task definition, container instances, and an ALB-fronted service
  • eks-quickstart: an EKS cluster with a node group, suitable as a starting point for Kubernetes workloads
  • eks-https-ingress: extends the EKS quickstart with private subnets, a NAT gateway, and HTTPS via the AWS Load Balancer Controller
  • eks-gitops-argocd: adds Argo CD and EBS-CSI persistent storage to the EKS cluster

Clone just the workbooks directory with a sparse checkout to keep the download small:

git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform-workbooks
Enter fullscreen mode Exit fullscreen mode

Each workbook has its own README that covers the variables, prerequisites, and teardown. The pattern for running any of them is the same:

export AWS_PROFILE=spinifex-<nodename>
cd docs/terraform-workbooks/<workbook-name>
tofu init
tofu apply
Enter fullscreen mode Exit fullscreen mode

The workbooks use OpenTofu (tofu) as the CLI, which is a drop-in replacement for terraform and is compatible with all Terraform configuration syntax and provider versions. If you prefer terraform, substitute it directly.

What transfers from your existing configs

The resource definitions, modules, variable files, outputs, and for_each/count patterns you've written against AWS all work against Spinifex. The provider override is the only change required at the HCL level, and the skip_* flags and endpoint variables in your provider block are the complete list of Spinifex-specific additions.

Full platform documentation, including the Terraform workbook references, is at docs.mulgadc.com. The source is on GitHub. Or sign up for the free sandbox to run Terraform plans against a live Spinifex endpoint before deploying on your own hardware.

Top comments (0)