cert-manager: All-in-One Kubernetes TLS Certificate Manager

Originally published at https://developer-friendly.blog on May 6, 2024.

Kubernetes is a great orchestration tool for managing your applications and all its dependencies. However, it comes with an extensible architecture and with an unopinionated approach to many of the day-to-day operational tasks.

One of these tasks is the management of TLS certificates. This includes issuing as well as renewing certificates from a trusted Certificate Authority. This CA may be a public internet-facing application or an internal service that needs encrypted communication between parties.

In this post, we will introduce the industry de-facto tool of choice for managing certificates in Kubernetes: cert-manager. We will walk you through the installation of the operator, configuring the issuer(s), and receiving a TLS certificate as a Kubernetes Secret for the Ingress or Gateway of your application.

Finally, we will create the Gateway CRD and expose an application securely over HTTPS to the internet.

If that gets you excited, hop on and let's get started!

Introduction

If you have deployed any reverse proxy in the pre-Kubernetes era, you might have, at some point or another, bumped into the issuance and renewal of TLS certificates. The trivial approach, back in the days as well as even today, was to use certbot¹. This command-line utility abstracts you away from the complexity of the underlying CA APIs and deals with the certificate issuance and renewal for you.

Certbot is created by the Electronic Frontier Foundation (EFF) and is a great tool for managing certificates on a single server. However, when you're working at scale with many applications and services, you will benefit from the automation and integration that cert-manager² provides.

cert-manager is a Kubernetes-native tool that extends the Kubernetes API with custom resources for managing certificates. It is built on top of the Operator Pattern³, and is a graduated project of the CNCF⁴.

With cert-manager, you can fetch and renew your TLS certificates behind automation, passing them along to the Ingress⁵ or Gateway⁶ of your platform to host your applications securely over HTTPS without losing the comfort of hosting your applications in a Kubernetes cluster.

With that introduction, let's kick off the installation of cert-manager.

Huge Thanks to You 🤗

If you're reading this, I would like to thank you for the time you spend on this blog 🌹. Whether this is your first time, or you've been here
before and have liked the content and its quality, I truly appreciate thetime you spend here.

As a token of appreciation, and to celebrate with you, I would like to
share the achievements of this blog over the course of ~11 weeks since its launch (the initial commit on Feb 13, 2024⁷).

• 10 posts published 📚
• 14k+ words written so far (40k+ including codes) 📝
• 2.5k+ views since the launch 👀
• 160+ clicks coming from search engines 🔍

Here are the corresponding screenshots:

I don't run ads on this blog (yet!? 🤔) and my monetization plan, as of the moment, is nothing! I may switch gear at some point; financial independence and doing this full-time makes me happy honestly ☺️.
But, for now, I'm just enjoying writing in Markdown format and seeing how Material for Mkdocs⁸ renders rich content from it.

If you are interested in supporting this effort, the GitHub Sponsors program, as well as the PayPal donation link are available at the bottom of all the pages in our website.

Greatly appreciate you being here and hope you keep coming back.

Pre-requisites

Before we start, make sure you have the following set up:

• A Kubernetes cluster. We have a couple of guides in our archive if you need help setting up a cluster:
  • Kubernetes the Hard Way
  • Lightweight K3s installation on Ubuntu
  • Azure AKS TF Module
• OpenTofu v1.7⁹
• Although not required, we will use FluxCD as a GitOps approach for our deployments. You can either follow along and use the Helm CLI instead, or follow our earlier guide for introduction to FluxCD.
• Optionally, External Secrets Operator installed. We will use it in this guide to store the credentials for the DNS01 challenge.
  • We have covered the installation of ESO in our last week's guide if you're interested to learn more: External Secrets Operator: Fetching AWS SSM Parameters into Azure AKS

Step 0: Installation

cert-manager comes with a first-class support for Helm chart installation.
This makes the installation rather straightforward.

As mentioned earlier, we will install the Helm chart using FluxCD CRDs.

# cert-manager/namespace.yml
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager

# cert-manager/repository.yml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: cert-manager
spec:
  interval: 60m
  url: https://charts.jetstack.io

# cert-manager/release.yml
apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: cert-manager
spec:
  chart:
    spec:
      chart: cert-manager
      sourceRef:
        kind: HelmRepository
        name: cert-manager
      version: v1.14.x
  interval: 30m
  maxHistory: 10
  releaseName: cert-manager
  targetNamespace: cert-manager
  timeout: 2m
  valuesFrom:
    - kind: ConfigMap
      name: cert-manager-config

Although not required, it is hugely beneficial to store the Helm values as it is in your VCS. This makes your future upgrades and code reviews easier.

helm repo add jetstack https://charts.jetstack.io
helm repo update jetstack
helm show values jetstack/cert-manager \
  --version v1.14.x > cert-manager/values.yml

# cert-manager/values.yml
# NOTE: truncated for brevity ...
# In a production setup, the whole file will be stored in VCS as is!
installCRDs: true

Additionally, we will use Kubernetes Kustomize¹⁰:

# cert-manager/kustomizeconfig.yml
nameReference:
  - kind: ConfigMap
    version: v1
    fieldSpecs:
      - path: spec/valuesFrom/name
        kind: HelmRelease

# cert-manager/kustomization.yml
configurations:
  - kustomizeconfig.yml

configMapGenerator:
  - files:
      - values.yaml=./values.yml
    name: cert-manager-config

resources:
  - namespace.yml
  - repository.yml
  - release.yml

namespace: cert-manager

Notice the namespace we are instructing Kustomization to place the resources in. The FluCD Kustomization CRD will be created in the flux-system namespace, while the Helm release itself is placed in the cert-manager namespace.

Ultimately, to create this stack, we will create a FluxCD Kustomization resource¹¹:

# cert-manager/kustomize.yml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cert-manager
  namespace: flux-system
spec:
  force: false
  interval: 10m0s
  path: ./cert-manager
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  wait: true

You may either advantage from the recursive reconciliation of FluxCD, add it to your root Kustomization or apply the resources manually from your command line.

kubectl apply -f cert-manager/kustomize.yml

Build Kustomization

A good practice is to build your Kustomization locally and optionally apply
them as a dry-run to debug any potential typo or misconfiguration.

  kustomize build ./cert-manager

And the output:

  apiVersion: v1
  kind: Namespace
  metadata:
    name: cert-manager
  ---
  apiVersion: v1
  data:
    values.yaml: |
      # NOTE: truncated for brevity ...
      # In a production setup, the whole file will be stored in VCS as is!
      installCRDs: true
  kind: ConfigMap
  metadata:
    name: cert-manager-config-8b8tf9hfb4
    namespace: cert-manager
  ---
  apiVersion: helm.toolkit.fluxcd.io/v2beta2
  kind: HelmRelease
  metadata:
    name: cert-manager
    namespace: cert-manager
  spec:
    chart:
      spec:
        chart: cert-manager
        sourceRef:
          kind: HelmRepository
          name: cert-manager
        version: v1.14.x
    interval: 30m
    maxHistory: 10
    releaseName: cert-manager
    targetNamespace: cert-manager
    timeout: 2m
    valuesFrom:
    - kind: ConfigMap
      name: cert-manager-config-8b8tf9hfb4
  ---
  apiVersion: source.toolkit.fluxcd.io/v1beta2
  kind: HelmRepository
  metadata:
    name: cert-manager
    namespace: cert-manager
  spec:
    interval: 60m
    url: https://charts.jetstack.io

Step 1.0: Issuer 101

In general, you can fetch your TLS certificate in two ways: either by verifying your domain using the HTTP01 challenge or the DNS01 challenge. Each have their own pros and cons, but both are just to make sure that you own the domain you're requesting the certificate for. Imagine a world where you could request a certificate for google.com without owning it! 😱

The HTTP01 challenge requires you to expose a specific path on your web server and asking the CA to send a GET request to that endpoint, expecting a specific file to be present in the response.

This is not always possible, especially if you're running a private service.

On a personal note, the HTTP01 feels like a complete hack to me. 😓

As such, in this guide, we'll use the DNS01 challenge. This challenge will create a specific DNS record in your nameserver. You don't specifically
have to manually do it yourself, as that is the whole point of automation that cert-manager will bring to the table.

For the DNS01 challenge, there are a couple of nameserver providers natively supported by cert-manager. You can find the list of supported providers on their website¹².

For the purpose of this guide, we will provide examples for two different nameserver providers: AWS Route53 and Cloudflare.

AWS services are the indudstry standard for many companies, and Route53 is one of the most popular DNS services (fame where it's due).

Cloudflare, on the other hand, is handling a significant portion of the internet's traffic and is known for its networking capabilities across the
globe.

If you have other needs, you won't find it too difficult to find support for your nameserver provider in the cert-manager documentation.

Step 1.1: AWS Route53 Issuer

The developer-friendly.blog domain is hosted in Cloudflare and to demonstrate the AWS Route53 issuer, we will make it so that a subdomain will be resolved
by a Route53 Hosted Zone. That way, we can instruct the cert-manager controller to talk to the Route53 API for record creation and domain verfication.

# hosted-zone/variables.tf
variable "root_domain" {
  type    = string
  default = "developer-friendly.blog"
}

variable "subdomain" {
  type    = string
  default = "aws"
}

variable "cloudflare_api_token" {
  type      = string
  nullable  = false
  sensitive = true
}

# hosted-zone/versions.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.47"
    }
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4.30"
    }
  }
}

provider "cloudflare" {
  api_token = var.cloudflare_api_token
}

# hosted-zone/main.tf
data "cloudflare_zone" "this" {
  name = var.root_domain
}

resource "aws_route53_zone" "this" {
  name = format("%s.%s", var.subdomain, var.root_domain)
}

resource "cloudflare_record" "this" {
  for_each = toset(aws_route53_zone.this.name_servers)

  zone_id = data.cloudflare_zone.this.id
  name    = var.subdomain
  type    = "NS"
  value   = each.value
  ttl     = 1

  depends_on = [
    aws_route53_zone.this
  ]
}

# hosted-zone/outputs.tf
output "hosted_zone_id" {
  value = aws_route53_zone.this.zone_id
}

output "name_servers" {
  value = aws_route53_zone.this.name_servers
}

To apply this stack we'll use OpenTofu.

We could've either separated the stacks to create the Route53 zone beforehand, or we will go ahead and target our resources separately from command line as
you see below.

export TF_VAR_cloudflare_api_token="PLACEHOLDER"
export AWS_PROFILE="PLACEHOLDER"

tofu plan -out tfplan -target=aws_route53_zone.this
tofu apply tfplan

# And now the rest of the resources

tofu plan -out tfplan
tofu apply tfplan

Why Applying Two Times?

The values in a TF for_each must be known at the time of planning, AKA, static values¹³.
And since that is not the case with aws_route53_zone.this.name_servers, we have to make sure to create the Hosted Zone first before passing its output to another resource.
We should have our AWS Route53 Hosted Zone created as you see in the screenshot
below.

Now that we have our Route53 zone created, we can proceed with the cert-manager configuration.

AWS IAM Role

We now need an IAM Role with enough permissions to create the DNS records to satisfy the DNS01 challenge¹⁴.

Make sure you have a good understanding of the
OpenID Connect, the technique we're employing in the trust relationship of the AWS IAM Role.

# route53-iam-role/variables.tf
variable "role_name" {
  type    = string
  default = "cert-manager"
}

variable "hosted_zone_id" {
  type        = string
  description = "The Hosted Zone ID that the role will have access to. Defaults to `*`."
  default     = "*"
}

variable "oidc_issuer_url" {
  type        = string
  description = "The OIDC issuer URL of the cert-manager Kubernetes Service Account token."
  nullable    = false
}

variable "access_token_audience" {
  type    = string
  default = "sts.amazonaws.com"
}

variable "service_account_name" {
  type        = string
  default     = "cert-manager"
  description = "The name of the service account."
}

variable "service_account_namespace" {
  type        = string
  default     = "cert-manager"
  description = "The namespace of the service account."
}

# route53-iam-role/versions.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.47"
    }
    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0"
    }
  }
}

# route53-iam-role/main.tf
data "aws_iam_policy_document" "iam_policy" {
  statement {
    actions = [
      "route53:GetChange",
    ]
    resources = [
      "arn:aws:route53:::change/${var.hosted_zone_id}",
    ]
  }

  statement {
    actions = [
      "route53:ChangeResourceRecordSets",
      "route53:ListResourceRecordSets",
    ]
    resources = [
      "arn:aws:route53:::hostedzone/${var.hosted_zone_id}",
    ]
  }

  statement {
    actions = [
      "route53:ListHostedZonesByName",
    ]
    resources = [
      "*",
    ]
  }
}

data "aws_iam_policy_document" "assume_role_policy" {
  statement {
    actions = [
      "sts:AssumeRoleWithWebIdentity"
    ]

    effect = "Allow"

    principals {
      type = "Federated"
      identifiers = [
        aws_iam_openid_connect_provider.this.arn
      ]
    }

    condition {
      test     = "StringEquals"
      variable = "${aws_iam_openid_connect_provider.this.url}:aud"

      values = [
        var.access_token_audience
      ]
    }

    condition {
      test     = "StringEquals"
      variable = "${aws_iam_openid_connect_provider.this.url}:sub"

      values = [
        "system:serviceaccount:${var.service_account_namespace}:${var.service_account_name}",
      ]
    }
  }
}

data "tls_certificate" "this" {
  url = var.oidc_issuer_url
}

resource "aws_iam_openid_connect_provider" "this" {
  url = var.oidc_issuer_url

  client_id_list = [
    var.access_token_audience
  ]

  thumbprint_list = [
    data.tls_certificate.this.certificates[0].sha1_fingerprint
  ]
}

resource "aws_iam_role" "this" {
  name = var.role_name
  inline_policy {
    name   = "${var.role_name}-route53"
    policy = data.aws_iam_policy_document.iam_policy.json
  }
  assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json
}

# route53-iam-role/outputs.tf
output "iam_role_arn" {
  value = aws_iam_role.this.arn
}

output "service_account_name" {
  value = var.service_account_name
}

output "service_account_namespace" {
  value = var.service_account_namespace
}

output "access_token_audience" {
  value = var.access_token_audience
}

tofu plan -out tfplan -var=oidc_issuer_url="KUBERNETES_OIDC_ISSUER_URL"
tofu apply tfplan

If you don't know what OpenID Connect is and what we're doing here, you might want to check out our ealier guides on the following topics:

• Establishing a trust relationship between bare-metal Kubernetes cluster and AWS IAM
• Same concept of trust relationship, this time between Azure AKS and AWS IAM

The gist of both articles is that we are providing a means for the two services to talk to each other securely and without storing long-lived credentials.

In essence, one service will issue the tokens (Kubernetes cluster), and the other will trust the tokens of the said service (AWS IAM).

Kubernetes Service Account

Now that we have our IAM role set up, we can pass that information to the cert-manager Deployment. This way the cert-manager will assume that role with the Web Identity Token flow¹⁵ (there are five flows in
total).

We will also create a ClusterIssuer CRD to be responsible for fetching the TLS certificates from the trusted CA.

# route53-issuer/variables.tf
variable "role_arn" {
  type    = string
  default = null
}

variable "kubeconfig_path" {
  type    = string
  default = "~/.kube/config"
}

variable "kubeconfig_context" {
  type    = string
  default = "k3d-k3s-default"
}

variable "field_manager" {
  type    = string
  default = "flux-client-side-apply"
}

variable "access_token_audience" {
  type    = string
  default = "sts.amazonaws.com"
}

variable "chart_url" {
  type    = string
  default = "https://charts.jetstack.io"
}

variable "chart_name" {
  type    = string
  default = "cert-manager"
}

variable "release_name" {
  type    = string
  default = "cert-manager"
}

variable "release_namespace" {
  type    = string
  default = "cert-manager"
}

variable "release_version" {
  type    = string
  default = "v1.14.x"
}

# route53-issuer/versions.tf
terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.29"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.13"
    }
  }
}

provider "kubernetes" {
  config_path    = var.kubeconfig_path
  config_context = var.kubeconfig_context
}

provider "helm" {
  kubernetes {
    config_path    = var.kubeconfig_path
    config_context = var.kubeconfig_context
  }
}

# route53-issuer/values.yml.tftpl
extraEnv:
  - name: AWS_ROLE_ARN
    value: ${sa_role_arn}
  - name: AWS_WEB_IDENTITY_TOKEN_FILE
    value: /var/run/secrets/aws/token

volumeMounts:
  - name: token
    mountPath: /var/run/secrets/aws
    readOnly: true

volumes:
  - name: token
    projected:
      sources:
        - serviceAccountToken:
            audience: ${sa_audience}
            expirationSeconds: 3600
            path: token

securityContext:
  fsGroup: 1001

# route53-issuer/main.tf
data "terraform_remote_state" "iam_role" {
  count = var.role_arn != null ? 0 : 1

  backend = "local"

  config = {
    path = "../route53-iam-role/terraform.tfstate"
  }
}

data "terraform_remote_state" "hosted_zone" {
  backend = "local"

  config = {
    path = "../hosted-zone/terraform.tfstate"
  }
}

locals {
  sa_audience = coalesce(var.access_token_audience, data.terraform_remote_state.iam_role[0].outputs.access_token_audience)
  sa_role_arn = coalesce(var.role_arn, data.terraform_remote_state.iam_role[0].outputs.iam_role_arn)
}

resource "helm_release" "cert_manager" {
  name       = var.release_name
  repository = var.chart_url
  chart      = var.chart_name
  version    = var.release_version
  namespace  = var.release_namespace

  reuse_values = true

  values = [
    templatefile("${path.module}/values.yml.tftpl", {
      sa_audience = local.sa_audience,
      sa_role_arn = local.sa_role_arn
    })
  ]
}

resource "kubernetes_manifest" "cluster_issuer" {
  manifest = yamldecode(<<-EOF
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: route53-issuer
    spec:
      acme:
        email: [e-mail]
        enableDurationFeature: true
        privateKeySecretRef:
          name: route53-issuer
        server: https://acme-v02.api.letsencrypt.org/directory
        solvers:
        - dns01:
            route53:
              hostedZoneID: ${data.terraform_remote_state.hosted_zone.outputs.hosted_zone_id}
              region: eu-central-1
  EOF
  )
}

# route53-issuer/outputs.tf
output "cluster_issuer_name" {
  value = kubernetes_manifest.cluster_issuer.manifest.metadata.name
}

tofu plan -out tfplan -var=kubeconfig_context="KUBECONFIG_CONTEXT"
tofu apply tfplan

If you're wondering why we're changing the configuration of the cert-manager Deployment with a new Helm upgrade, you will find an exhaustive discussion and my comment on the relevant GitHub issue¹⁶.

The gist of that conversation is that the cert-manager Deployment won't take into account the eks.amazonaws.com/role-arn annotation on its Service Account, as you'd see the External Secrets Operator would. It won't even consider using the ClusterIssuer.spec.acme.solvers[*].dns01.route53.role field for some reason! 🔫

That's why we're manually passing that information down to its AWS Go SDK¹⁷ using the official environment variables¹⁸.

This stack allows the cert-manager controller to talk to AWS Route53.

Notice that we didn't pass any credentials, nor did we have to create any IAM User for this communication to work. It's all the power of OpenID Connect and allows us to establish a trust relationship and never have to worry about any credentials in the client service. ✅

Is There a Simpler Way?

Sure there is. If you don't fancy OpenID Connect, there is always the option to pass the credentials around in your environment. That leaves you with the burden of having to rotate them every now and then, but if you're cool with that, there's nothing stopping you from going down that path. You also have the possibility of automating such rotation using less than 10 lines of code in any programming language of course.

All that said, I have to say that I consider this to be an implementation bug¹⁶; where cert-manager does not provide you with a clean interface to easily pass around IAM Role ARN. The cert-manager controller SHOULD be able to assume the role it is given with the web identity flow!

Regardless of such shortage, in this section, I'll provide you a simpler way around this.

Bear in mind that I do not recommend this approach, and wouldn't use it in my own environments either. 🤷

The idea is to use our previously deployed ESO and pass the AWS IAM User credentials to the cert-manager controller (easy peasy, no drama!).

# iam-user/variables.tf
variable "user_name" {
  type    = string
  default = "cert-manager"
}

variable "hosted_zone_id" {
  type        = string
  description = "The Hosted Zone ID that the role will have access to. Defaults to `*`."
  default     = "*"
}

# iam-user/versions.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.47"
    }
  }
}

# iam-user/main.tf
data "aws_iam_policy_document" "iam_policy" {
  statement {
    actions = [
      "route53:GetChange",
    ]
    resources = [
      "arn:aws:route53:::change/${var.hosted_zone_id}",
    ]
  }

  statement {
    actions = [
      "route53:ChangeResourceRecordSets",
      "route53:ListResourceRecordSets",
    ]
    resources = [
      "arn:aws:route53:::hostedzone/${var.hosted_zone_id}",
    ]
  }

  statement {
    actions = [
      "route53:ListHostedZonesByName",
    ]
    resources = [
      "*",
    ]
  }
}

resource "aws_iam_user" "this" {
  name = var.user_name
}

resource "aws_iam_access_key" "this" {
  user = aws_iam_user.this.name
}

resource "aws_ssm_parameter" "access_key" {
  for_each = {
    "/cert-manager/access-key" = aws_iam_access_key.this.id
    "/cert-manager/secret-key" = aws_iam_access_key.this.secret
  }

  name  = each.key
  type  = "SecureString"
  value = each.value
}

# iam-user/outputs.tf
output "iam_user_arn" {
  value = aws_iam_user.this.arn
}

output "iam_access_key_id" {
  value     = aws_iam_access_key.this.id
  sensitive = true
}

output "iam_access_key_secret" {
  value     = aws_iam_access_key.this.secret
  sensitive = true
}

And now let's create the corresponding ClusterIssuer, passing the credentials like a normal human being!

# route53-issuer-creds/variables.tf
variable "kubeconfig_path" {
  type    = string
  default = "~/.kube/config"
}

variable "kubeconfig_context" {
  type    = string
  default = "k3d-k3s-default"
}

variable "field_manager" {
  type    = string
  default = "flux-client-side-apply"
}

# route53-issuer-creds/versions.tf
terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.29"
    }
  }
}

provider "kubernetes" {
  config_path    = var.kubeconfig_path
  config_context = var.kubeconfig_context
}

# route53-issuer-creds/main.tf
data "terraform_remote_state" "hosted_zone" {
  backend = "local"

  config = {
    path = "../hosted-zone/terraform.tfstate"
  }
}

resource "kubernetes_manifest" "external_secret" {
  manifest = yamldecode(<<-EOF
    apiVersion: external-secrets.io/v1beta1
    kind: ExternalSecret
    metadata:
      name: route53-issuer-aws-creds
      namespace: cert-manager
    spec:
      data:
        - remoteRef:
            key: /cert-manager/access-key
          secretKey: awsAccessKeyID
        - remoteRef:
            key: /cert-manager/secret-key
          secretKey: awsSecretAccessKey
      refreshInterval: 5m
      secretStoreRef:
        kind: ClusterSecretStore
        name: aws-parameter-store
      target:
        immutable: false
  EOF
  )
}

resource "kubernetes_manifest" "cluster_issuer" {
  manifest = yamldecode(<<-EOF
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: route53-issuer
    spec:
      acme:
        email: [e-mail]
        enableDurationFeature: true
        privateKeySecretRef:
          name: route53-issuer
        server: https://acme-v02.api.letsencrypt.org/directory
        solvers:
        - dns01:
            route53:
              hostedZoneID: ${data.terraform_remote_state.hosted_zone.outputs.hosted_zone_id}
              region: eu-central-1
              accessKeyIDSecretRef:
                key: awsAccessKeyID
                name: route53-issuer-aws-creds
              secretAccessKeySecretRef:
                key: awsSecretAccessKey
                name: route53-issuer-aws-creds
  EOF
  )
}

# route53-issuer-creds/outputs.tf
output "external_secret_name" {
  value = kubernetes_manifest.external_secret.manifest.metadata.name
}

output "external_secret_namespace" {
  value = kubernetes_manifest.external_secret.manifest.metadata.namespace
}

output "cluster_issuer_name" {
  value = kubernetes_manifest.cluster_issuer.manifest.metadata.name
}

We're now done with the AWS issuer. Let's switch gear for a bit to create the Cloudflare issuer before finally creating a TLS certificate for our desired domain(s).

Step 1.2: Cloudflare Issuer

Since Cloudflare does not have native support for OIDC, we will have to pass an API token to the cert-manager controller to be able to manage the DNS records on our behalf.

That's where the External Secrets Operator comes into play, again. I invite you to take a look at our last week's guide if you haven't done so already.

We will use the ExternalSecret CRD to fetch an API token from the AWS SSM Parameter Store and pass it down to our Kubernetes cluster as a Secret resource.

Notice the highlighted lines.

# cloudflare-issuer/externalsecret.yml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: cloudflare-issuer-api-token
spec:
  data:
    - remoteRef:
        key: /cloudflare/api-token
      secretKey: cloudflareApiToken
  refreshInterval: 5m
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-parameter-store
  target:
    immutable: false

# cloudflare-issuer/clusterissuer.yml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cloudflare-issuer
spec:
  acme:
    email: [e-mail]
    enableDurationFeature: true
    privateKeySecretRef:
      name: cloudflare-issuer
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              key: cloudflareApiToken
              name: cloudflare-issuer-api-token
            email: [e-mail]

# cloudflare-issuer/kustomization.yml
resources:
  - externalsecret.yml
  - clusterissuer.yml

namespace: cert-manager

# cloudflare-issuer/kustomize.yml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cloudflare-issuer
  namespace: flux-system
spec:
  interval: 5m
  path: ./cloudflare-issuer
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  wait: true

kubectl apply -f cloudflare-issuer/kustomize.yml

That's all the issuers we aimed to create for today. One for AWS Route53 and another for Cloudflare.

We are now equipped with enough access in our Kubernetes cluster to just create the TLS certificate and never have to worry about how to verify their ownership.

With that promise, let's wrap this up with the easiest part! 😎

Step 2: TLS Certificate

You should have noticed by now that the root developer-friendly.blog will be resolved by Cloudflare as our initial nameserver. We also created a subdomain and a Hosted Zone in AWS Route53 to resolve the aws. subdomain using Route53 as its nameserver.

We can now fetch a TLS certificate for each of them using our newly created ClusterIssuer resource. The rest is the responsibility of the cert-manager to verify the ownership within the cluster through the DNS01 challenge and using the access we've provided it.

# tls-certificates/aws-subdomain.yml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: aws-developer-friendly-blog
spec:
  dnsNames:
  - '*.aws.developer-friendly.blog'
  issuerRef:
    kind: ClusterIssuer
    name: route53-issuer
  privateKey:
    rotationPolicy: Always
  revisionHistoryLimit: 5
  secretName: aws-developer-friendly-blog-tls

# tls-certificates/cloudflare-root.yml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: developer-friendly-blog
spec:
  dnsNames:
  - '*.developer-friendly.blog'
  issuerRef:
    kind: ClusterIssuer
    name: cloudflare-issuer
  privateKey:
    rotationPolicy: Always
  revisionHistoryLimit: 5
  secretName: developer-friendly-blog-tls

# tls-certificates/kustomization.yml
resources:
  - cloudflare-root.yml
  - aws-subdomain.yml

namespace: cert-manager

# tls-certificates/kustomize.yml
resources:
  - cloudflare-root.yml
  - aws-subdomain.yml

namespace: cert-manager

kubectl apply -f tls-certificates/kustomize.yml

It'll take less than a minute to have the certificates issued and stored as Kubernetes Secrets in the same namespace as the cert-manager Deployment.

If you would like the certificates in a different namespace, you're better off creating Issuer instead of ClusterIssuer.

The final result will have a Secret with two keys: tls.crt and tls.key. This will look similar to what you see below.

---
- apiVersion: cert-manager.io/v1
  kind: Certificate
  metadata:
    name: aws-developer-friendly-blog
  namespace: cert-manager
  spec:
    dnsNames:
      - "*.aws.developer-friendly.blog"
    issuerRef:
      kind: ClusterIssuer
      name: route53-issuer
    privateKey:
      rotationPolicy: Always
    revisionHistoryLimit: 5
    secretName: aws-developer-friendly-blog-tls
  status:
    conditions:
      - lastTransitionTime: "2024-05-04T05:44:12Z"
        message: Certificate is up to date and has not expired
        observedGeneration: 1
        reason: Ready
        status: "True"
        type: Ready
    notAfter: "2024-07-30T04:44:12Z"
    notBefore: "2024-05-04T04:44:12Z"
    renewalTime: "2024-06-29T04:44:12Z"
---
apiVersion: v1
data:
  tls.crt: ...truncated...
  tls.key: ...truncated...
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: "*.aws.developer-friendly.blog"
    cert-manager.io/certificate-name: aws-developer-friendly-blog
    cert-manager.io/common-name: "*.aws.developer-friendly.blog"
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: ClusterIssuer
    cert-manager.io/issuer-name: route53-issuer
    cert-manager.io/uri-sans: ""
  labels:
    controller.cert-manager.io/fao: "true"
  name: aws-developer-friendly-blog-tls
  namespace: cert-manager
type: kubernetes.io/tls

Step 3: Use the TLS Certificates in Gateway

At this point, we have the required ingredients to host an application within cluster and exposing it securely through HTTPS into the world.

That's exactly what we aim for at this step. But, first, let's create a Gateway CRD that will be the entrypoint to our cluster. The Gateway can be thought of as the sibling of Ingress resource, yet more handsome, more successful, more educated and more charming¹⁹.

The key point to keep in mind is that the Gateway API doesn't come with the implementation. Infact, it is unopinionated about the implementation and you can use any networking solution that fits your needs and has support for it.

In our case, and based on the personal preference and tendency of the author 😇, we'll use Cilium as the networking solution, both as the CNI, as well as the implementation for our Gateway API.

We have covered the Cilium installation before, but, for the sake of completeness, here's the way to do it²⁰.

# cilium/playbook.yml
- name: Bootstrap the Kubernetes cluster
  hosts: localhost
  gather_facts: false
  become: true
  environment:
    KUBECONFIG: ~/.kube/config
  vars:
    helm_version: v3.14.4
    kube_context: k3d-k3s-default
  tasks:
    - name: Install Kubernetes library
      ansible.builtin.pip:
        name: kubernetes<30
        state: present
    - name: Install helm binary
      ansible.builtin.shell:
        cmd: "{{ lookup('ansible.builtin.url', 'https://git.io/get_helm.sh', split_lines=false) }}"
        creates: /usr/local/bin/helm
      environment:
        DESIRED_VERSION: "{{ helm_version }}"
    - name: Install Kubernetes gateway CRDs
      kubernetes.core.k8s:
        src: "{{ item }}"
        state: present
        context: "{{ kube_context }}"
      loop:
        - https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
        - https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml
    - name: Install cilium
      block:
        - name: Add cilium helm repository
          kubernetes.core.helm_repository:
            name: cilium
            repo_url: https://helm.cilium.io
        - name: Install cilium helm release
          kubernetes.core.helm:
            name: cilium
            chart_ref: cilium/cilium
            namespace: kube-system
            state: present
            chart_version: 1.15.x
            kube_context: "{{ kube_context }}"
            values:
              gatewayAPI:
                enabled: true
              kubeProxyReplacement: true
              encryption:
                enabled: true
                type: wireguard
              operator:
                replicas: 1

And now, let's create the Gateway CRD.

# gateway/gateway.yml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: developer-friendly-blog
spec:
  gatewayClassName: cilium
  listeners:
    - allowedRoutes:
        namespaces:
          from: All
      name: http
      port: 80
      protocol: HTTP
    - allowedRoutes:
        namespaces:
          from: All
      name: https
      port: 443
      protocol: HTTPS
      tls:
        certificateRefs:
          - group: ""
            kind: Secret
            name: developer-friendly-blog-tls
            namespace: cert-manager
          - group: ""
            kind: Secret
            name: aws-developer-friendly-blog-tls
            namespace: cert-manager
        mode: Terminate

Notice that we did not create the gatewayClassName. It comes as battery-included with Cilium. You can find the GatewayClass as soon as Cilium installation completes with the following command:

kubectl get gatewayclass

GatewayClass is to Gateway as IngressClass is to Ingress.

Also note that we are passing the TLS certificates to this Gateway we have created earlier. That way, the gateway will terminate and offload the SSL/TLS encryption and your upstream service will receive plaintext traffic.

However, if you have set up your mTLS the way we did with Wireguard encryption (or any other mTLS solution for that matter), node-to-node and/or pod-to-pod communications will also be encrypted.

# gateway/http-to-https-redirect.yml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: https-redirect
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: developer-friendly-blog
    namespace: cert-manager
    sectionName: http
  rules:
  - filters:
    - requestRedirect:
        scheme: https
        statusCode: 301
      type: RequestRedirect
    matches:
    - path:
        type: PathPrefix
        value: /

Though not required, the above HTTP to HTTPS redirect allows you to avoid accepting any plaintext HTTP traffic on your domain.

# gateway/kustomization.yml
resources:
  - gateway.yml
  - http-to-https-redirect.yml

namespace: cert-manager

# gateway/kustomize.yml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: gateway
  namespace: flux-system
spec:
  interval: 5m
  path: ./gateway
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  wait: true

kubectl apply -f gateway/kustomize.yml

Step 4: HTTPS Application

That's all the things we aimed to do today. At this point, we can create our HTTPS-only application and expose it securely to the wild internet!

# app/deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-server
  template:
    metadata:
      labels:
        app: echo-server
    spec:
      containers:
        - envFrom:
            - configMapRef:
                name: echo-server
          image: ealen/echo-server
          name: echo-server
          ports:
            - containerPort: 80
              name: http
          securityContext:
            capabilities:
              add:
                - NET_BIND_SERVICE
              drop:
                - ALL
            readOnlyRootFilesystem: true
      securityContext:
        runAsGroup: 1000
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault

# app/service.yml
apiVersion: v1
kind: Service
metadata:
  name: echo-server
spec:
  ports:
    - name: http
      port: 80
      targetPort: http
  type: ClusterIP

# app/httproute.yml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echo-server
spec:
  hostnames:
  - echo.developer-friendly.blog
  - echo.aws.developer-friendly.blog
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: developer-friendly-blog
    namespace: cert-manager
    sectionName: https
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: echo-server
      port: 80
      weight: 1
    filters:
    - responseHeaderModifier:
        set:
        - name: Strict-Transport-Security
          value: max-age=31536000; includeSubDomains; preload
      type: ResponseHeaderModifier
    matches:
    - path:
        type: PathPrefix
        value: /

# app/configs.env
PORT=80
LOGS__IGNORE__PING=false
ENABLE__HOST=true
ENABLE__HTTP=true
ENABLE__REQUEST=true
ENABLE__COOKIES=true
ENABLE__HEADER=true
ENABLE__ENVIRONMENT=false
ENABLE__FILE=false

# app/kustomization.yml
resources:
  - deployment.yml
  - service.yml
  - httproute.yml

images:
  - name: ealen/echo-server
    newTag: 0.9.2

configMapGenerator:
  - name: echo-server
    envs:
      - configs.env

replacements:
  - source:
      kind: Deployment
      name: echo-server
      fieldPath: spec.template.metadata.labels
    targets:
      - select:
          kind: Service
          name: echo-server
        fieldPaths:
          - spec.selector
        options:
          create: true
  - source:
      kind: ConfigMap
      name: echo-server
      fieldPath: data.PORT
    targets:
      - select:
          kind: Deployment
          name: echo-server
        fieldPaths:
          - spec.template.spec.containers.[name=echo-server].ports.[name=http].containerPort

namespace: default

# app/kustomize.yml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: app
  namespace: flux-system
spec:
  interval: 5m
  path: ./app
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  wait: true

kubectl apply -f app/kustomize.yml

That's everything we had to say for today. We can now easily access our application as follows:

curl -v https://echo.developer-friendly.blog -sSo /dev/null

or...

curl -v https://aws.echo.developer-friendly.blog -sSo /dev/null

...truncated...
*  expire date: Jul 30 04:44:12 2024 GMT
...truncated...

Both will show that the TLS certificate is present. signed by a trusted CA, is valid and matches the domain we're trying to access. 🎉

You shall see the same expiry date on your certificate if accessing as follows:

kubectl get certificate \
  -n cert-manager \
  -o jsonpath='{.items[*].status.notAfter}'

2024-07-30T04:44:12Z

As you can see, the information we get from the publicly available certificate as well as the one we get internally from our Kubernetes cluster are the same down to the second. 💪

Conclusion

These days, I am never spinning up a Kubernetes cluster without having cert-manager installed on it as its day 1 operation task. It's such a life-saver tool to have in your toolbox and you can rest assured that the TLS certificates in your cluster are always up-to-date and valid.

If you ever had to worry about the expiry date of your certificates before, those days are behind you and you can benefit a lot by employing the cert-manager operator in your Kubernetes cluster. Use it to its full potential and you shall be served greatly.

Hope you enjoyed reading this material.

Until next time 🫡, ciao 🤠 and happy hacking! 🦀
🐧 🐳