DEV Community: Chris Phillipson

Crossplane and Terraform

Chris Phillipson — Sun, 29 Jan 2023 00:56:37 +0000

Today, I asked ChatGPT a few questions so I might understand whether it would be advantageous to use Crossplane with Terraform.

Here's the exchange...

Please compare and contrast the use of Terraform and Crossplane when used to create and destroy managed public cloud resources.

ChatGPT

Terraform and Crossplane are both Infrastructure as Code (IAC) tools that can be used to create and destroy managed public cloud resources. However, they have some key differences in their design and use-cases.

Terraform is a tool developed by HashiCorp that allows users to define and provision infrastructure resources using HashiCorp Configuration Language (HCL). It supports a wide range of cloud providers, including AWS, Azure, and Google Cloud, as well as on-premises and other infrastructure. Terraform is focused on provisioning and managing resources on a specific cloud provider.

Crossplane, on the other hand, is an open-source control plane for multi-cloud infrastructure that allows users to define and provision infrastructure resources using Kubernetes Custom Resource Definitions (CRDs). It supports multiple cloud providers, but also provides an abstraction layer over them, allowing users to provision resources using a consistent API regardless of the underlying cloud provider.

In summary, Terraform is a cloud-agnostic tool that helps in provisioning and managing resources on a specific cloud provider, whereas Crossplane is an open-source control plane for multi-cloud infrastructure that provides a consistent API for provisioning resources across multiple cloud providers.

Please contrast how Crossplane and Terraform manage state. What happens when Terraform state is lost? What happens to managed resources when the cluster Crossplane is deployed into is destroyed?

ChatGPT

Both Terraform and Crossplane manage the state of the infrastructure they are provisioning, but they do it in different ways.

Terraform uses a local or remote state file to store the current state of the infrastructure it is managing. This state file is used to ensure that the infrastructure is in the desired state after a Terraform apply command is executed. The state file can be stored locally or remotely, such as in a cloud object storage service like S3. If the state file is lost or becomes out of sync, Terraform can become unable to determine the correct state of the infrastructure, which can lead to unexpected changes or failures when applying Terraform configurations. To mitigate this issue, Terraform provides a number of state management features, such as state locking and state versioning, to help ensure the state file remains accurate.

Crossplane, on the other hand, stores the state of the infrastructure it is managing in the Kubernetes cluster where it is deployed. This state is stored in Kubernetes Custom Resource Definitions (CRDs) and can be managed using the Kubernetes API. If the Kubernetes cluster where Crossplane is deployed is destroyed, the state of the infrastructure managed by Crossplane will also be lost. However, Crossplane also has built-in recovery mechanisms to ensure that the state of the infrastructure is restored when the cluster is recreated.

In summary, Terraform uses a local or remote state file to manage the state of the infrastructure it provisions, whereas Crossplane stores the state of the infrastructure it provisions in the Kubernetes cluster where it is deployed. If the state file is lost, Terraform can become unable to determine the correct state of the infrastructure, whereas if the cluster Crossplane is deployed into is destroyed, the state of the infrastructure managed by Crossplane will also be lost, but it has built-in recovery mechanisms.

Can Crossplane detect whether a resource has already been created without state? Likewise, can Terraform detect whether a resource has been created without state?

ChatGPT

Both Crossplane and Terraform can detect whether a resource has already been created without state, but they do it in different ways.

Crossplane can detect whether a resource has already been created without state by querying the Kubernetes API for the existence of the resource. Because Crossplane stores the state of the infrastructure it is managing in the Kubernetes cluster where it is deployed, it can use the Kubernetes API to check if the resource already exists. If the resource already exists, Crossplane can retrieve its current state and compare it to the desired state defined in the Crossplane resource definition.

Terraform can also detect whether a resource has already been created without state by querying the cloud provider's API for the existence of the resource. Terraform can also check if the resource already exists in the state file. If the resource already exists, Terraform can retrieve its current state and compare it to the desired state defined in the Terraform configuration file.

In summary, Crossplane can detect whether a resource has already been created without state by querying the Kubernetes API, whereas Terraform can detect whether a resource has already been created without state by querying the cloud provider's API and state file.

Can you show me how to configure Terraform to update the state of an existing resource when that resource is not currently managed by Terraform?

ChatGPT

Yes, you can configure Terraform to update the state of an existing resource when that resource is not currently managed by Terraform. This can be done using the terraform import command.

Here is an example of how to use the terraform import command to update the state for an existing AWS S3 bucket:

# First, determine the resource type and the unique identifier for the resource you want to import
aws s3 ls # This command will show all the buckets in your AWS account
# Let's say you want to import an existing bucket named "my-bucket"
# Next, use the terraform import command to import the existing resource into Terraform's state
terraform import aws_s3_bucket.my_bucket my-bucket

The above command imports the existing S3 bucket named "my-bucket" into Terraform's state. This will allow Terraform to know about the existence of the resource and treat it as a managed resource going forward.

It is important to note that running the import command will not change the state of the resource, it just makes Terraform aware of the resource, so you will need to run terraform plan and terraform apply after running the import command to make sure that the resource state matches with the state defined in the Terraform configuration file.

Also, You may need to check if the provider you are using is already defined in your Terraform configuration file. If not, you need to define it before running the import command.

Is it more advantageous to use Crossplane versus Terraform for the reason that Crossplane will continuously reconcile and prevent configuration drift?

ChatGPT

Crossplane and Terraform have different advantages and use cases.

Crossplane is designed to provide a unified interface for provisioning and managing multiple cloud providers and on-premises resources. It uses Kubernetes Custom Resource Definitions (CRDs) to define the desired state of the infrastructure and automatically reconciles the state of the infrastructure with the desired state defined in the CRDs. This allows for continuous monitoring and correction of any drift that may occur between the actual state of the infrastructure and the desired state. This makes Crossplane particularly useful for organizations that are looking to provision and manage resources across multiple clouds and on-premises environments using a consistent set of APIs and abstractions.

On the other hand, Terraform is a tool specifically designed for provisioning and managing infrastructure. It uses a configuration language called HashiCorp Configuration Language (HCL) to define the desired state of the infrastructure. Terraform can provision resources in multiple cloud providers and on-premises environments as well, but it doesn't have built-in mechanisms to continuously reconcile and prevent configuration drift like Crossplane. However, it can detect drift between the current state and the desired state and generates a plan that shows what changes are needed to bring the infrastructure in the desired state.

In summary, Crossplane provides continuous monitoring and correction of any drift that may occur between the actual state of the infrastructure and the desired state, making it particularly useful for organizations looking to provision and manage resources across multiple clouds and on-premises environments using a consistent set of APIs and abstractions. Terraform is a powerful tool for provisioning and managing infrastructure in multiple cloud providers and on-premises environments, but it does not have built-in mechanisms for continuous drift reconciliation like Crossplane.

Are you able to articulate the benefits of using Crossplane with its Terraform provider?

ChatGPT

Using Crossplane with its Terraform provider can provide several benefits:

Consistent API for provisioning and managing resources: Crossplane provides a consistent API for provisioning and managing resources across multiple cloud providers and on-premises environments, regardless of whether the resources are managed using Terraform or other provisioning tools.
Automated drift detection and correction: By using Crossplane's built-in mechanisms for continuous monitoring and correction of any drift that may occur between the actual state of the infrastructure and the desired state, you can ensure that your resources are always in the desired state, even if they were initially provisioned using Terraform.
Centralized management: With Crossplane, you can centrally manage and provision resources across multiple cloud providers and on-premises environments, which can help to simplify the management and organization of your infrastructure.
Multi-cloud resource provisioning: Crossplane's Terraform provider allows you to provision resources across multiple clouds in a consistent way, meaning that you can use the same Terraform configuration to provision resources in AWS, Azure, GCP, and other cloud providers.
Reusability: With Crossplane, you can reuse the same Terraform modules and providers that you use for provisioning resources in the public cloud, in your on-premises environments as well.
Compliance and Governance: Crossplane can enforce compliance and governance policies by automating the detection of drift and correcting it, across all the resources provisioned by Terraform.

In summary, using Crossplane with its Terraform provider can help organizations to have a consistent, centralized and automated way to provision, manage and monitor resources in multiple cloud providers and on-premises environments, with the added benefits of compliance and governance enforcement.

Yelling at my laptop - Part 1

Chris Phillipson — Fri, 27 Jan 2023 07:00:41 +0000

Preface

This post chronicles my attempt at installing a commercial, secure software supply-chain platform on top of a public cloud provider's managed Kubernetes-as-service offering by not explicitly following public documentation.

First, let me share a personal bias. I don't have the best recall. Being asked to execute a complex task that I may have completed a few weeks or months ago heightens my anxiety. Particularly, if I'm under a time constraint. So, if I know I'm going to have to do something repeatedly, I'm going to automate it. I do this to reduce toil and avoid potential mistakes.

On the choice of a technology stack

Every DIY developer I know forms opinions on the choice of tech stack to aid in value delivery. They may or may not have the authority and autonomy to exercise those opinions in an enterprise setting, where the choice of toolset, frameworks and automation may be dictated by any number of reasons (e.g., feasibility, contractual, community, financial, regulatory). Therefore, when I settle on a technology stack to accomplish a task, I want to think about how I'd position its applicability in that context.

To accomplish the task of provisioning public cloud resources, I chose Terraform. It ticks several boxes, not the least that I have several years of experience with it. I need to be able to create and destroy cloud environments hosting a footprint of multiple Kubernetes clusters. At a minimum that consists of a virtual network and subnets, a container registry, one or more clusters, one or more DNS zones, blob storage, and a secrets manager instance.

To work with a cluster, of course, I'll need the kubectl client. To make my life easier I'll install a few krew plugins like ctx, reap, and view-secret. To build, tag, pull and push container images for test purposes, I'll use a combination of docker and pack - in anger. Lastly, I'll rely on a complement of other sharp tools, like yq, kapp, and ytt.

Provisioning cloud resources

With time I've acquired and developed skills with HCL, authoring and consuming Terraform modules. I've often found it to be the case that when I want to stitch something together myself someone else has authored a coarse-grained module on Github or the Terraform Registry that's designed in such a way that I can easily leverage and/or extend it. (It's particularly gratifying when I pick a module where large communities have evolved to maintain and consume it).

Creating an Artifact Registry instance

With that bit of background, let's focus on the creation of a Google Artifact Registry. Artifact registries allow you to host many types of artifacts; for my purposes, I needed to create one to host container images. The first thing you need to grok (which I failed to understand early on, wasting huge amounts of time) is how artifacts are organized for storage and retrieval. The public documentation available speaks to how you create, authenticate, then push and pull container images.

So, after glancing at that I set about designing a Terraform module to lifecycle-manage an Artifact registry instance and a repository within it.

main.tf

data "external" "env" {
  program = ["${path.module}/env.sh"]
}

data "google_kms_key_ring" "terraform" {
  name = var.keyring
  location = var.location
}

data "google_kms_crypto_key" "terraform" {
  name = "terraform"
  key_ring = data.google_kms_key_ring.terraform.id
}

resource "google_artifact_registry_repository" "instance" {
  for_each = toset(var.repository_names)
  location = var.location
  kms_key_name = data.google_kms_crypto_key.terraform.id
  repository_id = each.key
  description = "OCI image repository"
  format = "DOCKER"
}

env.sh

#!/bin/sh

# env.sh

# Change the contents of this output to get the environment variables
# of interest. The output must be valid JSON, with strings for both
# keys and values.
cat <<EOF
{
  "google_service_account_key": "$GOOGLE_SERVICE_ACCOUNT_KEY"
}
EOF

variables.tf

variable "project" {
  description = "A valid Google project identifier"
  sensitive = true
}

variable "repository_names" {
  type = list(string)
  description = "Specifies the names of repositories that will be created within Google Cloud Artifact Registry"
  default = ["tanzu"]
}

variable "location" {
  description = "Google Cloud Artifact Registry Repository and Google Cloud KMS keyring locations"

  validation {
    condition = contains(["us-west1", "us-west2", "us-east1", "us-central1", "europe-north1", "europe-west1", "europe-southwest1", "asia-east1", "asia-south1", "asia-northeast3", "australia-southeast2"], var.location)
    error_message = "Valid values for Google Cloud Artifact Registry locations are (us-west1, us-west2, us-east1, us-central1, europe-north1, europe-west1, europe-southwest1, asia-east1, asia-south1, asia-northeast3, australia-southeast2)."
  }

  default = "us-west2"
}

variable "keyring" {}

providers.tf

provider "google" {
  project = var.project
}

versions.tf

terraform {

  required_version = ">= 0.14.0"

  required_providers {
    google = {
      source = "hashicorp/google"
      version = ">= 4.33.0"
    }
  }
}

outputs.tf

output "admin_username" {
  value = "_json_key_base64"
}

output "admin_password" {
  description = "The base64-encoded password associated with the Container Registry admin account"
  value = base64encode(data.external.env.result["google_service_account_key"])
  sensitive = true
}

output "endpoint" {
  value = "${var.location}-docker.pkg.dev"
}

terraform.tfvars

project = "REPLACE_ME"
repository_names = ["tanzu"]
location = "us-west2"
keyring = "REPLACE_ME"

create-artifact-registry.sh

#!/bin/bash

terraform init -upgrade
terraform validate
terraform plan -out terraform.plan
terraform apply -auto-approve -state terraform.tfstate terraform.plan

destroy-artifact-registry.sh

#!/bin/bash

terraform destroy -auto-approve
rm -Rf .terraform .terraform.lock.hcl terraform.tfstate terraform.tfstate.backup terraform.log terraform.plan

Note the use of scripts. I don't want to have to think too hard about the sequence of Terraform commands to issue to create and destroy this cloud resource.

Also, importantly, we're only creating one repository named tanzu within the registry instance.

The outputs from this module allow us to authenticate, push/pull container images to/from the registry instance.

Working with the Artifact Registry instance repository

If we have an instance available, we want to be able to interact with it, right?

Let's quickly run through the sequence of commands to pull a public image, tag it, and then push it to the registry instance repository.

E.g.,

#! Authenticate
echo "cat $HOME/.ssh/terraform-sa.json" | base64 -w0 | docker login -u _json_key_base64 --password-stdin us-west2-docker.pkg.dev
#! Pull a public image from Dockerhub
docker pull busybox
#! Tag the image
docker tag busybox us-west2-docker.pkg.dev/fe-cphillipson/tanzu/busybox
#! Push the image to the Artifact Registry instance repository
docker push us-west2-docker.pkg.dev/fe-cphillipson/tanzu/busybox

Provisioning a cloud environment

To be able to install Tanzu Application Platform, we'll need multiple resources provisioned. I've curated a set of Terraform modules to do that, here. At a minimum, consider executing these modules: iam, project, kms, registry, virtual-network, cluster, main-dns, and child-dns.

Installing Tanzu Application Platform

While I provided a link to install TAP (in online-mode), we're not going to follow those instructions. As I mentioned earlier, I wanted to put myself in a position to repeat these steps with ease and minimal toil and friction.

I am setting the stage to share a series of posts with you about how I assembled automation targeting multiple cloud providers.

With that in mind, we're going to stare at some YAML, because who doesn't like to do that, huh?

But before we do that we need to satisfy some prerequisites:

Register for a new account on Tanzu Network
Accept the EULAs for VMWare Tanzu Application Platform.

Next, we need to create a namespace and install two controllers into the cluster. (You will already have administrator access if you used the Terraform cluster module above).

#! Create namespace where TAP will be installed
kubectl create namespace tap-install

#! Install kapp-controller
#! @see https://carvel.dev/kapp-controller/docs/v0.43.2/install/
kubectl apply -f https://github.com/vmware-tanzu/carvel-kapp-controller/releases/latest/download/release.yml

#! Install secretgen-controller
#! @see https://github.com/carvel-dev/secretgen-controller/blob/develop/docs/install.md
kubectl apply -f https://github.com/carvel-dev/secretgen-controller/releases/latest/download/release.yml

Now, let's create some directories

mkdir tap-install
cd tap-install
mkdir base
mkdir config
mkdir -p profiles/iterate

And here beginneth a wave of templated YAML files.

Place the following files in the tap-install/ base folder.

rbac.yml

#@ load("@ytt:data", "data")
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tap-default-sa
  namespace: #@ data.values.tap.namespace
  annotations:
    kapp.k14s.io/change-group: tap-install/rbac
    kapp.k14s.io/change-rule: "delete after deleting tap-install/tap"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tap-default-role
  annotations:
    kapp.k14s.io/change-group: tap-install/rbac
    kapp.k14s.io/change-rule: "delete after deleting tap-install/tap"
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tap-default-role-binding
  annotations:
    kapp.k14s.io/change-group: tap-install/rbac
    kapp.k14s.io/change-rule: "delete after deleting tap-install/tap"
subjects:
- kind: ServiceAccount
  name: tap-default-sa
  namespace: #@ data.values.tap.namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tap-default-role

container-image-registry-secret.yml

#@ load("@ytt:data", "data")
#@ load("@ytt:json", "json")
---
#@ def crc_config():
#@ return {
#@ "auths": {
#@ data.values.tap.credentials.registry.host: {
#@ "email": data.values.tap.credentials.registry.email,
#@ "username": data.values.tap.credentials.registry.username,
#@ "password": data.values.tap.credentials.registry.password
#@ }
#@ }
#@ }
#@ end
---
apiVersion: v1
kind: Secret
metadata:
  name: container-registry-credentials
  namespace: #@ data.values.tap.namespace
  annotations:
    kapp.k14s.io/change-rule: "delete after deleting tap"
type: kubernetes.io/dockerconfigjson
stringData:
  .dockerconfigjson: #@ json.encode(crc_config())
---
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretExport
metadata:
  name: container-registry-credentials
  namespace: #@ data.values.tap.namespace
spec:
  toNamespaces:
  - '*'

dev-namespace.yml

#@ load("@ytt:data", "data")
---
#@ if data.values.tap.devNamespace != "default" and data.values.tap.devNamespace != "":
apiVersion: v1
kind: Namespace
metadata:
  name: #@ data.values.tap.devNamespace
#@ end
---
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretImport
metadata:
  name: container-registry-credentials
  namespace: #@ data.values.tap.devNamespace
spec:
  fromNamespace: #@ data.values.tap.namespace
---
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretImport
metadata:
  name: tanzu-network-credentials
  namespace: #@ data.values.tap.devNamespace
spec:
  fromNamespace: #@ data.values.tap.namespace
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: #@ data.values.tap.devNamespace
secrets:
  - name: container-registry-credentials
imagePullSecrets:
  - name: container-registry-credentials
  - name: tanzu-network-credentials
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: #@ "{}-{}".format(data.values.tap.devNamespace, "permit-deliverable")
  namespace: #@ data.values.tap.devNamespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: deliverable
subjects:
  - kind: ServiceAccount
    name: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: #@ "{}-{}".format(data.values.tap.devNamespace, "permit-workload")
  namespace: #@ data.values.tap.devNamespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: workload
subjects:
  - kind: ServiceAccount
    name: default

tanzu-network-secret.yml

#@ load("@ytt:data", "data")
#@ load("@ytt:base64", "base64")
#@ load("@ytt:json", "json")
---
#@ def tn_config():
#@ return {
#@ "auths": {
#@ data.values.tap.credentials.tanzuNet.host: {
#@ "username": data.values.tap.credentials.tanzuNet.username,
#@ "password": data.values.tap.credentials.tanzuNet.password
#@ }
#@ }
#@ }
#@ end
---
apiVersion: v1
kind: Secret
metadata:
  name: tanzu-network-credentials
  namespace: #@ data.values.tap.namespace
  annotations:
    kapp.k14s.io/change-rule: "delete after deleting tap"
type: kubernetes.io/dockerconfigjson
stringData:
  .dockerconfigjson: #@ json.encode(tn_config())
---
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretExport
metadata:
  name: tanzu-network-credentials
  namespace: #@ data.values.tap.namespace
spec:
  toNamespaces:
  - '*'

tap-package-repo.yml

#@ load("@ytt:data", "data")
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
  name: tanzu-tap-repository
  namespace: #@ data.values.tap.namespace
  annotations:
    kapp.k14s.io/change-group: tap-install/tap-repo
spec:
  fetch:
    imgpkgBundle:
      image: #@ "{}/tanzu-application-platform/tap-packages:{}".format(data.values.tap.credentials.tanzuNet.host, data.values.tap.version)
      secretRef:
        name: tanzu-network-credentials

tap-package-install.yml

#@ load("@ytt:data", "data")
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  name: tap
  namespace: #@ data.values.tap.namespace
  annotations:
    kapp.k14s.io/change-group: tap-install/tap
    kapp.k14s.io/change-rule.0: "upsert after upserting tap-install/rbac"
    kapp.k14s.io/change-rule.1: "upsert after upserting tap-install/tap-repo"
    packaging.carvel.dev/downgradable: ""
spec:
  packageRef:
    refName: tap.tanzu.vmware.com
    versionSelection:
      constraints: #@ str(data.values.tap.version)
      prereleases: {}
  serviceAccountName: tap-default-sa
  values:
  - secretRef:
      name: tap-values

Place the following files in the tap-install/profiles/ iterate folder.

tbs-full-dependencies-package-repo.yml

#@ load("@ytt:data", "data")
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
  name: tbs-full-deps-repository
  namespace: #@ data.values.tap.namespace
  annotations:
    kapp.k14s.io/change-group: tap-install/tbs-full-deps-repo
spec:
  fetch:
    imgpkgBundle:
      image: #@ "{}/tanzu-application-platform/full-tbs-deps-package-repo:{}".format(data.values.tap.credentials.tanzuNet.host, data.values.buildService.version)
      secretRef:
        name: tanzu-network-credentials

tbs-full-dependencies-package-install.yml

#@ load("@ytt:data", "data")
#@ load("@ytt:yaml", "yaml")
---
#@ def stack_config():
  stack_configuration: #@ data.values.tap.stack_configuration
#@ end
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  name: tbs-full-dependencies
  namespace: #@ data.values.tap.namespace
  annotations:
    kapp.k14s.io/change-group: tap-install/tbs-full-deps
    kapp.k14s.io/change-rule.0: "upsert after upserting tap-install/tap"
    kapp.k14s.io/change-rule.1: "upsert after upserting tap-install/tbs-full-deps-repo"
    packaging.carvel.dev/downgradable: ""
spec:
  packageRef:
    refName: full-tbs-deps.tanzu.vmware.com
    versionSelection:
      constraints: #@ str(data.values.buildService.version)
      prereleases: {}
  serviceAccountName: tap-default-sa
  values:
  - secretRef:
      name: tbs-full-dependencies
---
apiVersion: v1
kind: Secret
metadata:
  name: tbs-full-dependencies
  namespace: #@ data.values.tap.namespace
stringData:
  tbs-full-dependencies-secrets.yml: #@ yaml.encode(stack_config())

Place the following files in the tap-install/ config directory.

tap-iterate-values-template.yml

#@ load("@ytt:data", "data")
#@ load("@ytt:yaml", "yaml")
---
#@ def config():
profile: iterate

ceip_policy_disclosed: true

shared:
  ingress_domain: #@ data.values.tap.domains.main

buildservice:
  kp_default_repository: #@ "{}/{}".format(data.values.tap.credentials.registry.host, data.values.tap.registry.repositories.buildService)
  kp_default_repository_username: #@ data.values.tap.credentials.registry.username
  kp_default_repository_password: #@ data.values.tap.credentials.registry.password
  tanzunet_username: #@ data.values.tap.credentials.tanzuNet.username
  tanzunet_password: #@ data.values.tap.credentials.tanzuNet.password
  exclude_dependencies: true
  stack_configuration: #@ data.values.tap.stack_configuration

#! supply_chain is pinned to basic as we want fast feedback in inner loop development
supply_chain: basic

ootb_supply_chain_basic:
  cluster_builder: full
  registry:
    server: #@ data.values.tap.credentials.registry.host
    repository: #@ data.values.tap.registry.repositories.ootbSupplyChain
  gitops:
    ssh_secret: #@ data.values.tap.supply_chain.gitops.ssh_secret

scanning:
  metadataStore:
    url: ""

metadata_store:
  ns_for_export_app_cert: #@ data.values.tap.devNamespace
  app_service_type: ClusterIP

image_policy_webhook:
  allow_unmatched_tags: true

contour:
  envoy:
    service:
      type: LoadBalancer

cnrs:
  domain_name: #@ data.values.tap.domains.knative
  domain_template: #@ "{}-{}-{}".format("{{.Name}}", data.values.tap.domains.suffix, "{{.Namespace}}.{{.Domain}}")

appliveview_connector:
  backend:
    sslDisabled: false
    ingressEnabled: true
    host: #@ "appliveview.{}".format(data.values.tap.domains.main)

#@ end
---
apiVersion: v1
kind: Secret
metadata:
  name: tap-values
  namespace: #@ data.values.tap.namespace
type: Opaque
stringData:
  values.yml: #@ yaml.encode(config())

config.yml

#@data/values

#@overlay/match-child-defaults missing_ok=True
---
buildService:
  version: "1.9.0"

tap:
  version: "1.4.0"
  namespace: tap-install
  devNamespace: development
  catalogs: []

  registry:
    repositories:
      buildService: REPLACE_WITH_YOUR_GOOGLE_PROJECT_ID/tanzu/build-service
      ootbSupplyChain: REPLACE_WITH_YOUR_GOOGLE_PROJECT_ID/tanzu/supply-chain

  domains:
    main: tap.REPLACE_WITH_YOUR_DOMAIN
    knative: apps.tap.REPLACE_WITH_YOUR_DOMAIN
    tapGui: tap-gui.tap.REPLACE_WITH_YOUR_DOMAIN
    suffix: iterate

  #! Change to "jammy-only" if you want to install Tanzu Application Platform with Ubuntu 22.04 (Jammy) as the only available stack
  stack_configuration: "default"

  supply_chain:
    cluster_builder: full
    #! choices below are: [go-git, libgit2]
    git_implementation: go-git
    gitops:
      ssh_secret: ""

sensitive-config.yml

#@data/values

#@overlay/match-child-defaults missing_ok=True
---
tap:
  credentials:
    registry:
      host: REPLACE_WITH_GOOGLE_REGION-docker.pkg.dev
      username: _json_key_base64
      password: REPLACE_WITH_BASE64_ENCODED_GOOGLE_SERVICE_ACCOUNT_JSON_FILE_CONTENTS
    tanzuNet:
      host: registry.tanzu.vmware.com
      username: REPLACE_WITH_TANZUNET_USERNAME
      password: REPLACE_WITH_TANZUNET_PASSWORD

Phew! That was quite a wave. Now we'll marry the configuration with the templated manifest to generate a concrete manifest named tap-iterate-values.yml using ytt. (Make sure you are still in the tap-install directory).

ytt -f config > tap-iterate-values.yml

Go ahead and inspect the result, it's just a Secret.

cat tap-iterate-values.yml

Let's deploy it

kubectl apply -f tap-iterate-values.yml

To cap it all off we'll use the combination of ytt and kapp to install Tanzu Application Platform activating the iterate profile.

kapp deploy -a tap-iterate -f <(ytt -f config/config.yml -f config/sensitive-config.yml -f base -f profiles/iterate) -c -y

Execute the following command to see that all components were installed successfully. You may have to do this multiple times if you're impatient. A typical installation completes within 20 minutes. (You'll know that when the DESCRIPTION column shows Reconcile succeeded for all installed components).

kubectl get app -A

Final thoughts

We spent a fair amount of time cultivating HCL and YAML to drive the desired outcome of automated provisioning of public cloud resources and installation of TAP. And we were exposed to some sharp tools. However, exercising the building blocks above still leads to toil.

What if we were asked to create multiple environments? No doubt, we'd start crying.

And what about collaboration? I would want the system of record to be a Git repository, not my laptop. I don't want to tie up my laptop compute either (when provisioning cloud resources with Terraform), so I would consider leveraging a self-service continuous integration/delivery engine.

What else might we do to improve upon this approach?

Stay tuned for my next post.

Installing cert-manager webhook for OCI and Let's Encrypt ClusterIssuer on an OKE cluster

Chris Phillipson — Tue, 24 Jan 2023 02:00:45 +0000

If you're an operator maintaining a footprint of Kubernetes clusters, then you already know that there's more to provisioning and granting access to a cluster. For a cluster to be useful for an end consumer an operator will usually install, enable and configure several other components. One such component is cert-manager. Its purpose is to issue certificates from a variety of supported sources. Consulting the docs, you might learn about the ACME issuer type and the DNS01 challenge validation method. Then depending on your IaaS requirements, you will find a limited set of supported providers and webhooks.

And now we come to the reason why I'm writing this post. If you're contemplating installing cert-manager an OKE cluster on OCI, you'll want to employ a webhook. However, it is not as straightforward as you might think. Uninitiated, you might visit this link and follow the instructions to install it. The implementation has not kept up with the API changes over successive releases of cert-manager. So, you would be forced to install a pre-v1.x version of cert-manager. Not a great option as you'd be left open to many security vulnerabilities. I want to share how I built, tested and published an up-to-date version of the aftermentioned, here.

Upgrading the implementation

I decided to target compatibility with cert-manager v1.10 or better, so that the updated webhook would be functional on Kubernetes clusters, versions 1.20 through 1.26.

As I reviewed the Gitlab repository where the original webhook implementation and Helm chart reside, I noticed that another developer had made an effort to upgrade the webhook to be compatible with cert-manager 1.8 in this fork. So, I cloned that fork and began to make updates.

I upgraded the Golang version in go.mod to 1.19 and decided to target Kubernetes version 1.24 in Makefile. I also updated the args in Dockerfile to be

ARG GOLANG_VERSION=1.19.4
ARG ALPINE_VERSION=3.17

First, I upgraded the libraries

go get -u all
go mod tidy

Then, I attempted to build the container image that would be employed by the Helm chart.

docker build -t pacphi/cert-manager-webhook-oci .

Where to host this image? In a public container image registry repository. I chose an OCI registry instead of Dockerhub because I (and others who might want to consume my pre-built image) won't get rate-limited on image pull requests.

Lastly, I authored a BaSH script to help me remember the sequence of steps to create a repo (if it doesn't already exist), build and tag the image, obtain an auth token, authenticate to the registry, and finally push the image.

Inspecting the image

I put myself in the shoes of a potential consumer. Could they successfully pull the image with no authentication credentials from the public repository I had just created? And how might they learn about what's truly inside the image?

To pull the image, they'd execute

docker pull phx.ocir.io/axyd58snjxbf/cert-manager-webhook-oci:latest

To explore the layers of the image, they could use dive

dive phx.ocir.io/axyd58snjxbf/cert-manager-webhook-oci:latest

Upgrading the Helm chart

The chart configuration required only one update. In apiservice.yaml, I needed to revert all occurrences of v1 back to v1alpha. Why? The acme.d-n.be apiservice is not discoverable otherwise.

Installing and uninstalling the Helm chart

Instructions in the README of the original repository left much to be desired. So, I authored another BaSH script to help me consistently install the prerequisites and the webhook itself. And what about teardown? There's a script for that too.

Testing it out

Before you can leverage the install and uninstall scripts, you'll need to:

* add a DNS zone in OCI

* add a VCN in OCI

* provision a cluster

* update the install script's environment variables

That last bullet I might improve upon soon so that you don't need to edit the script itself.

Here's a fast and inexpensive way to test

#! Create a single-node cluster on your workstation with kind; @see https://kind.sigs.k8s.io/
kind create cluster
#! Setup access to your Oracle Cloud Infastructure account
oci setup config
#! Clone the repo where the updated webhook implementation resides
git clone https://github.com/paphi/cert-manager-webhook-oci
#! Change directories into the scripts directory
cd cert-manager-webhook-oci/scripts
#! Edit and save updates to install-cert-manager-webhook-oci.sh
#! Then run this script to install
./install-cert-manager-webhook-oci.sh
#! To validate the installation...
kubectl get pods -A
kubectl get apiservice | grep acme.d-n.be
kubectl get clusterissuer
kubectl get cert -A
kubectl get secret -A
#! To uninstall, run this script
./uninstall-cert-manager-webhook-oci.sh
#! To destroy the cluster
kind delete cluster

But if you want to test this out against an OKE cluster, you could follow this tutorial.