DEV Community: Jello

Using external-dns on Kubernetes - Cloudflare

Jello — Wed, 23 Aug 2023 10:43:50 +0000

When the load balancer's IP address changes, our goal is to avoid the need for manual updates on Cloudflare. Instead, we aim to employ a monitoring service that will automatically ensure the continuity of our DNS records despite these IP changes.

Initially, let's set up external DNS. To do this, you will need the following information from Cloudflare:

API KEY
ZONE ID

We need to create a secret that external-dns can use to access the Cloudflare API.

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: default
type: Opaque
stringData:
   api-token: ${CLOUDFLARE_API_TOKEN}

RBAC Configuration

This RBAC configuration grants the external-dns ServiceAccount permissions to interact with namespaces and specific resources in the gateway.networking.k8s.io API group.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get","watch","list"]
- apiGroups: ["gateway.networking.k8s.io"]
  resources: ["gateways","httproutes","grpcroutes","tlsroutes","tcproutes","udproutes"]
  verbs: ["get","watch","list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: default

Deploying external-dns

In the deployment replace the $ZONE_ID with the zone ID from cloudflare
secretKeyRef refers to the cloudflare-api-token-secret that we created earlier
Pay attention to --source=gateway-httproute as this flag defines what resource will be monitored for DNS changes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.k8s.io/external-dns/external-dns:v0.13.5
        imagePullPolicy: Always
        args:
        - --source=gateway-httproute
        - --zone-id-filter=$ZONE_ID
        - --provider=cloudflare
        - --cloudflare-proxied
        - --policy=upsert-only
        - --log-level=debug
        env:
          - name: CF_API_TOKEN
            valueFrom:
              secretKeyRef:
                name: cloudflare-api-token-secret
                key: api-token

Verify that the deployment is ready and now we can annotate the services that we want to update the dns records for.

The HTTPRoute resource defines a routing configuration for incoming HTTP requests with a specific host and path. When a request matches the defined conditions, it will be directed to the test-app1 backend service running on port 80. This allows for fine-grained control over how incoming traffic is directed within a Kubernetes cluster.
⚠️ Important Note: Make sure that the CNAME/A name is not already created so external dns can create the record and take ownership from the start.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: test-app-1
spec:
  parentRefs:
  - name: fine-ops-gateway
    sectionName: https
  hostnames:
    - "fe-1.fine-ops.com"
  rules:
    - matches:
      - path:
          type: PathPrefix
          value: /
        method: GET
      backendRefs:
      - name: test-app1
        port: 80

Verify that the route was created be executing kubectl get httproutes

After we have applied the HTTP resource lets see if external-dns has created a new DNS record for us.Looking at the logs of the pod kubectl logs external-dns-548f86c768-82szq we can see that

external-dns dected the HTTP Resource and created a new A record:
Double-checking also from the DNS cloudflare dashboard :

Issuing wildcard certificates on Kubernetes Gateway API (Part 1/2)

Jello — Wed, 23 Aug 2023 10:42:39 +0000

What you will need to follow along:

A domain name and a cloudflare api token with the following permissions:
Helm

First install cert-manager

#Add the helm repo
helm repo add jetstack https://charts.jetstack.io

# You can remove the experimental support if you don't need it 
helm install cert-manager jetstack/cert-manager --version v1.12.3 \
    --namespace cert-manager \
    --set installCRDs=true \
    --create-namespace \
    --set "extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}"

Validating the deployment by executing

kubectl get pods -n cert-manager

should produce the following output

We need cluster issuer to be able to create certificates in our cluster

First we need to create scecret with the cloudflare token in the cert manager space

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
   api-token: ${CLOUDFLARE_API_TOKEN}

Then we can create the cluster issuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cloudflare-domain-issuer
spec:
  acme:
    email: email-to-register-domain
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        cloudflare:
          email: cloudflare-email
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token

kubectl get cluster issuer

if the cluster issuer is ready then we can issue our wildcard certificate
secretName: fine-ops: Specifies the name of the Kubernetes Secret where the generated certificate and private key will be stored.

Note: secretName specifies the name of the Kubernetes Secret where the generated certificate and private key will be stored.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: certificate-fine-ops
  namespace: default
spec:
  dnsNames:
    - "*.fine-ops.com"
  secretName: fine-ops
  issuerRef:
    name: cloudflare-domain-issuer
    kind: ClusterIssuer

kubectl get cr

Here you have it :)

Using the certificate in the kubernetes Gateway API

Issuing wildcard certificates on Kubernetes Gateway API (Part 2/2)

Jello — Wed, 23 Aug 2023 10:37:43 +0000

If you have followed part1 you should should have cert-manager already running.

All the files are available on github: https://github.com/angjelo/cloud-infra

Let's see how we can build the following infrustrcture by using cilium and the gateway api.

Initially we need to create a gateway class and specify the controller that our gateway will be using, I am using cilium on my cluster that's why I have set controllerName: io.cilium/gateway-controller

apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: cillium-gateway-class
spec:
  controllerName: io.cilium/gateway-controller

After you have successfully created the gateway controller lets define the gateway which will be our entrypoint.
Few things:

The annotations field allows you to attach additional metadata to the resource. In this case, an annotation is used to specify the cluster issuer for certificates issued by Cert-Manager (a Kubernetes add-on for certificate management)
cert-manager.io/cluster-issuer:cloudflare-domain-issuer refers to the cluster-issuer that we created in part1
The secret used in the field certificateRefs is defined in the certificate.yml

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: fine-ops-gateway
  annotations:
    cert-manager.io/cluster-issuer: cloudflare-domain-issuer
spec:
  gatewayClassName: cillium-gateway-class
  listeners:
    - name: https
      hostname: "*.fine-ops.com"
      port: 443
      protocol: HTTPS
      tls:
        mode: Terminate
        certificateRefs:
        - kind: Secret
          name: fine-ops

if we execute kubectl get svc then we shoulld see a public IP assigned to our gateway, if you are on aws usually you will get an ELB address that you can use to point to point your CNAME cloudflare

⚠️ Important Note: if you delete and recreate the gateway the load balancer will populate a new ip which you need to update manually on cloudflare

Now that we have defined our main entry point lets create our

tls-route
service
deployment for test-app1

This will be the nginx app deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app1
  namespace: default
  labels:
    app: test-app1
spec:
  replicas: 3
  selector:
    matchLabels:
      app: test-app1
  template:
    metadata:
      labels:
        app: test-app1
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        imagePullPolicy: Always

kubectl apply -f test-app1-deployment.yml

To view if the pods are deployed successfully kubectl get pods

Now is time to create a service to expose the port internally in kubernetes:

apiVersion: v1
kind: Service
metadata:
  name: test-app1
  namespace: default
  labels:
    app: test-app1
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: test-app1

kubectl get svc to see if the service was created

Se we can see that test-app1 has been exposed to an internal IP. Now it's time create the tls-route and attach it to the service that we exposed earlier

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: test-app-1
spec:
  parentRefs:
  - name: fine-ops-gateway
    sectionName: https
  hostnames:
    - "fe-1.fine-ops.com"
  rules:
    - matches:
      - path:
          type: PathPrefix
          value: /
        method: GET
      backendRefs:
      - name: test-app1
        port: 80

Lets break down some keypoints in the yaml:

parentRefs refers to the gateway name that we created earlier (fine-ops-gateway)
We are referencing the service created earlier for test-app1 validate that the route was created: kubectl get httproutes

Now that we have have created our http route, we can try to access the service on https://fe-1.fine-ops.com (make sure you have created an A record that points the Loadbalancer IP of the cilium gateway)

Now that we have deployed fe-1.fine-ops.com it time to deploy fe-2.fine-ops.com

Let's start again by creating the second deployment, similar to the previous nginx deployment but the app name is different this time

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app2
  namespace: default
  labels:
    app: test-app2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app2
  template:
    metadata:
      labels:
        app: test-app2
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        imagePullPolicy: Always

After verifying that the pods are running it is time to create the service to internally expose the port to kubernetes

apiVersion: v1
kind: Service
metadata:
  name: test-app2
  namespace: default
  labels:
    app: test-app2
spec:
  ports:
  - port: 90
    name: http
    targetPort: 80
  selector:
    app: test-app2

Not it is time to create the TLS route for fe-2.fine-ops.com

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-app-2
spec:
  parentRefs:
  - name: fine-ops-gateway
    namespace: default
  hostnames:
    - "fe-2.fine-ops.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
      method: GET
    backendRefs:
    - name: test-app2
      port: 90

kubectl get httproutes should display these routes

Lastly we need to add a new A for fe-2.fine-ops.com record to point to the load balancer IP,this is how the records should look like in cloudflare

Now you can visit fe-2.yourdomain.com

Secure builds on Kubernetes with Kaniko (AWS ECR)

Jello — Wed, 23 Aug 2023 10:27:52 +0000

Kaniko is a suitable choice for scenarios where security, isolation, compatibility with container orchestration platforms, and efficient resource utilization are important factors. It provides an alternative to traditional Docker builds, particularly in environments where running Docker daemons might not be optimal or secure.

All the files are included in gitlab

Prerequisites:

Kubernetes Cluster
Helm Installed
gitlab-registration token
IAM User from AWS with full ECR ACCESS

Lets start by adding the AWS credentials as variables into our repository

The helm repository will enable us to use the gitlab-runner chart

helm repo add gitlab https://charts.gitlab.io

Gitlab runner config yaml

A deeper overview of the gitlab configuration is located in the official repo ["https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/values.yaml"]

gitlabUrl: https://gitlab.com
runnerRegistrationToken: $REGISTRATION_TOKEN
concurrent: 2
rbac:
  create: true
  name: gitlab-runner
runners:
  tags: kube,aws
  runUntagged: true
  imagePullPolicy: if-not-present

gitalb-runner/values.yml contains the values to overwrite in the helm repostiry
tags: kube,aws can be anything that makes sense to you
runnerRegistrationToken should be obtained from the CI/CD section.

Create a namespace for gitlab: kubectl create ns gitlab

Using helm to instal gitlab-runner

helm install --namespace gitlab gitlab-runner -f values.yml gitlab/gitlab-runner

kubectl get pods -n gitlab should have active gitlab pods

Now if you navigate to the runners section in your gitlab, there should be a new runner registered:

For the project build you can use the Dockerfile & index.html from my gitlab-repoisitory

.gitlab-ci.yml

To further understand kaniko please reference to the documentation kaniko Gitlab

stages:
  - build

variables:
  AWS_REGION: eu-north-1
  REGISTRY: "${AWS_ACCOUNT_ID}.dkr.ecr.eu-north-1.amazonaws.com"

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"credsStore\":\"ecr-login\",\"credHelpers\":{\"$REGISTRY/contoso\":\"ecr-login\"}}" > /kaniko/.docker/config.json
    - >-
      /kaniko/executor
      --context "${CI_PROJECT_DIR}" 
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile" 
      --build-arg AWS_REGION=$AWS_REGION 
      --build-arg AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID 
      --build-arg AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY 
      --destination "${REGISTRY}/contoso:${CI_COMMIT_TAG:-$CI_COMMIT_REF_SLUG}"
  tags:
    - aws,kube

"{\"credsStore\":\"ecr-login\",\"credHelpers\":{\"$REGISTRY/contoso\":\"ecr-login\"}}":

credsStore: This specifies the Docker credentials store to be used for authentication. In this case, it's set to "ecr-login", in which the Amazon Elastic Container Registry (ECR) authentication mechanism is being used.
credHelpers: Is an object that defines custom credential helpers for specific Docker registries. In this case, a custom helper is being set up for the $REGISTRY/contoso registry.

Once we push gitlab-ci.yml and trigger the pipeline we see the build is running, it Could take up to 30 secs for the runner to pick up the job

You can see the full pipelinepipeline log:

Navigate to the ECR repository to confirm the image build

Now that you have the pipeline all working start experimenting and see what you can do :)

Cilium with Kubernetes Gateway API on Azure Cloud (BYOCNI)

Jello — Wed, 16 Aug 2023 19:20:18 +0000

Spinning up the AKS Cluster

I selected the cheapest VM type that I found to keep the costs to minimum for this project

You will need AKS Preview enabled for the commands to work properly

We are planning to use cilium without kube-proxy, the flag --kube-proxy-config kube-proxy.json wil disable it upon creation.

kube-proxy json config

{
  "enabled": false,
  "mode": "IPVS",
  "ipvsConfig": {
    "scheduler": "LeastConnection",
    "TCPTimeoutSeconds": 900,
    "TCPFINTimeoutSeconds": 120,
    "UDPTimeoutSeconds": 300
  }
}

In azure-cli or terminal execute the following command (make sure you are using an already authenticated user):

export NAME="cilium-sw"
export AZ_RESOURCEGROUP="resource group"
export AZ_LOCATION="ylocation"

az aks create \
    -n "${NAME}" \
    --network-plugin=none \
    -l "${AZ_LOCATION}" \
    -g "${AZ_RESOURCEGROUP}" \
    --kube-proxy-config="kube-proxy.json" \
    --enable-cluster-autoscaler \
    --min-count=1 \
    --max-count=2 \
    --no-wait \
    --node-count=1 \
    --node-vm-size="Standard_B2s" \
    --load-balancer-sku="basic" \
    --generate-ssh-keys

Connecting to the AKS Cluster

az aks get-credentials --resource-group cilium-eu-1_group --name cilium-eu

Coredns & metrics services will be stuck because there is no CNI plugin installed in your cluster, you can verify that by running:

kubectl get pods -n kube-system

Installing resources that are need for Gateway API support

Before we install cilium as we are planning to use the Kubernetes Gateway API we need to isntall following compoents to kubernetes (also you can reference the officials docs: Gateway API Cilium)

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml

Installing Cilium

There are different ways to install cilium according to this articles:

$ git clone https://github.com/cilium/cilium
$ cd cilium
# Then we will use helm to install cilium with the latest changes #from the main branch
$ cilium install --chart-directory ./install/kubernetes/cilium \
    --set kubeProxyReplacement=true \
    --set gatewayAPI.enabled=true
    --set azure.resourceGroup=${AZ_RESOURCEGROUP}

Upon successful installation you should see the following:

kubectl get pods -n kube-system

Now your cilium cluster is ready :)