DEV Community: Robin Cher

Automating Kong Konnect Configuration with Terraform

Robin Cher — Fri, 07 Jun 2024 06:37:19 +0000

Introduction

HashiCorp built Terraform on top of a plug-in system, where vendors can build their own extensions to Terraform. These extensions are called “providers.” Providers map the declarative configuration into the required API interactions, ensuring that the desired state is met. They act as a bridge between Terraform and a third-party API.

Kong has always placed developer experience as top priority, and building a terraform provider is a no-brainer since its widely adopted by the community at large

For today walkthrough, we will attempt to create a Control Plane, Service , Route and a Rate Limit Plugin in Kong Konnect. Kong Konnect is a hybrid saas platform where the control plane is hosted/managed by Kong, and customer will deploy Data Plane(proxy) on their own environment.

Getting Started

Ensure you have

Terraform CLI installed
Kong Konnect Control Plane Access

First ,lets create a auth.tf that will configure your Kong Konnect tf provider, and a personal access token for authentication with Kong Konnect.

You can generate a access token by navigating to the top right, click on** Personal Access Token*, and then * Generate Token**



# auth.tf
# Configure the provider to use your Kong Konnect account
terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
      version = "0.2.5"
    }
  }
}

provider "konnect" {
  personal_access_token = "kpat_xxxx"
  server_url            = "https://au.api.konghq.com"
}

Subsequently, lets create the resources declarative file



#main.tf

# Create a new Control Plane
resource "konnect_gateway_control_plane" "tfdemo" {
  name         = "Terraform Control Plane"
  description  = "This is a sample description"
  cluster_type = "CLUSTER_TYPE_HYBRID"
  auth_type    = "pinned_client_certs"

  proxy_urls = [
    {
      host     = "example.com",
      port     = 443,
      protocol = "https"
    }
  ]
}

# Configure a service and a route that we can use to test
resource "konnect_gateway_service" "httpbin" {
  name             = "HTTPBin"
  protocol         = "https"
  host             = "httpbin.org"
  port             = 443
  path             = "/"
  control_plane_id = konnect_gateway_control_plane.tfdemo.id
}

resource "konnect_gateway_route" "anything" {
  methods = ["GET"]
  name    = "Anything"
  paths   = ["/anything"]

  strip_path = false

  control_plane_id = konnect_gateway_control_plane.tfdemo.id
  service = {
    id = konnect_gateway_service.httpbin.id
  }
}

resource "konnect_gateway_plugin_rate_limiting" "my_rate_limiting_plugin" {
  enabled = true
  config = {
    minute = 5
    policy = "local"
  }

  protocols        = ["http", "https"]
  control_plane_id = konnect_gateway_control_plane.tfdemo.id
  route = {
    id = konnect_gateway_route.anything.id
  }
}

Run a terraform plan to validate what will be build



terraform plan

You should have the following file in the directory

Run the terraform apply to commit the resources



terraform apply

If everything went well, you should see a freshly created Control plane with a sample Service and Route attached with a Rate Limit Plugin

Summary

With a Konnect TF provider, customers can leverage on existing CI/CD pipeline to run Kong's api configuration automatically and consistently across different environment. DevEX is something Kong will be focusing on, and do expect more toolings from Kong in the coming months!

Resources

Kong Konnect TF provider - https://github.com/Kong/terraform-provider-konnect
Kong Konnect - https://docs.konghq.com/konnect/

Automating Kong API Gateway deployment with Flux

Robin Cher — Thu, 27 Apr 2023 13:30:11 +0000

Introduction

In recent years, GitOps has created an industry shift in how configuration change is managed. GitOps elevates the source control repository to the source of truth for configuration change management, and makes the repository the central hub of change control. The benefits of following this development paradigm are many, for example, GitOps helps:

Improve Collaboration
Increase deployment reliability, stability and frequency
Decrease deployment time and reduce human error
Improve compliance and auditing
and many others…

For these reasons, many engineering organizations are implementing GitOps, including in control of deployments to their API Gateway.

One of the most popular tools in this space is Flux, which we'll be using today.For this post, we will be setting up Flux to demonstrate how one can deploy Kong in a GitOps fashion.

What is Flux?

Let's start by defining what Flux is. Flux is a tool used to keep your Kubernetes clusters in sync with sources of configuration, such as Git repositories, and also automate updates to your configuration when there is new code to deploy. The declarative nature of Flux means that Kong configurations can be written as a set of facts directly within the source code, and Git is the "single source of truth". Essentially, Flux focuses on managing your infrastructure and platform, the way that a developer already is familiar with, using git to commit code changes.

What is Kong?

Kong is the world's most adopted API Gateway that lets you secure, manage and extend APIs or microservices. Get started with Kong API Gateway and learn why it’s one of the best API Gateway in the industry.

Architecture

Now that we understand what Flux is, let's dive into what our architecture looks like when using Flux. Below, we'll find a diagram that summarizes how Flux (with its CRDs) manages Kong using GitOps.

As one can see from the diagram above, the Platform Engineer has one responsibility: to push code to the repository. It is at this point that the whole GitOps flow starts and triggers a number of actions that will finish with a release of an API (in our case). Let's dive into it further.
Tech Stack and Tooling

Note: You will incur some cost in running public cloud resources, do remember to tear it down upon this exercise.

For this walkthrough, these are the tools that we will be using:

Flux v2 (and its CLI)
Github
Access to Amazon Elastic Kubernetes Service (EKS)
kubectl
aws cli
eksctl

Prerequisites

To get started, we need to make sure that we have our stack set up, so we'll perform the following steps:

Fork this Repository, and clone it locally - https://github.com/robincher/kong-flux-gitops. This will be the working directory for us to execute commands for this exercise
Generate a Github’s Personal Access Tokens (PAT) - Refer to the Github Guide for details

How to Deploy Kong using Flux

Set up the EKS Cluster**

To get started, let's create an EKS cluster with eksctl



eksctl create cluster --name Kong-GitOps-Test-Cluster  --version 1.23 --region <preferred-aws-region>  --without-nodegroup

Subsequently, let create a node group for the cluster



eksctl create nodegroup --cluster Kong-GitOps-Test-Cluster --name Worker-NG  --region <preferred-aws-region>  --node-type t3.medium --nodes 1 --max-pods-per-node 50

After the cluster is completely set up, let's ensure you are able to access the kube-api by running the following:



aws eks update-kubeconfig --region <preferred-aws-region> --name Kong-GitOps-Test-Cluster

Let check if the cluster is up and you have access to it



kubectl get nodes

There should be one node up and running



NAME                                               STATUS   ROLES    AGE    VERSION
ip-192-168-49-64.ap-southeast-1.compute.internal   Ready    <none>   6d2h   v1.23.7

Creating a Remote RDS Postgres (For Kong Control Plane)

Generally we will advise customers to use a Managed Postgres when running Kong in production-like environments.

Remember to pre-create an initial database. When using a remote database like RDS, Kong will not automatically initialize the database.

Next, create a secret for the DB password



kubectl create secret generic kong-db-password --from-literal=postgresql-password=xxxxx -n kong

Let’s test the connectivity from the EKS cluster to Postgres by first creating a temporary postgres pod



kubectl run -i --tty --rm debug --image=postgres --restart=Never -- sh

Subsequently., we can run a test command to check the connectivity.



psql --host=postgres.internal.somewhere --port=5432 --username=konger --password --dbname=kong

Lastly, we can create an ExternalName for the postgres host.



kubectl create service externalname  kong-db-postgresql  --external-name postgres.internal.somewhere -n kong

Deploying Kong via HelmRepository and HelmRelease

Now we should have our cluster properly bootstrapped.

Now it's time to briefly explain the configurations. For our experiment, we will deploy Kong using Helm. Flux supports helm deployments via their HelmRepository and HelmRelease CRDs.

Let’s go into the folder you just forked and clone locally

cd ~/yourpath/kong-flux-gitops

HelmRepository defines the source where Flux will attempt to pull the helm charts from.

If you already forked from the repository, you will see a HelmRepository CRD pre-configured for you to pull Kong’s helm charts. At this stage, you do not need to modify anything for this file



#sources/kong.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: kong
spec:
  url: https://charts.konghq.com
  interval: 10m

Let’s look at the HelmRelease CRD that will we will be updating, and we will go through the configuration step by step



cat ~/yourpath/kong-flux-gitops/platform/kong-gw/release/yaml

Understanding HelmRelease CRD

From Flux’s Official documentation: HelmRelease defines a Flux’s resource for controller driven reconciliation of Helm releases via Helm actions such as install, upgrade, test, uninstall, and rollback. As such, we need to allow Flux to understand what to deploy and configure for Kong.

The first step is to specify the chart that will be pulled for the deployment under spec.chart, and indicate the HelmRepository that was created in the previous step, and the corresponding chart version.



#platform/kong-gw/release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: kong
spec:
  interval: 5m
  chart:
    spec:
      chart: kong
      version: 2.15.3
      sourceRef:
        kind: HelmRepository
        name: kong
        namespace: flux-system
      interval: 1m

Subsequently, the next section is spec.values where you will indicate all possible helm values from Kong Official Helm Chart which is located in Kong's official Github repository, here.

So, without further ado, let’s start off by setting the image repository and the tag for the kong container image that we will be deploying. The result can be shown below:



#platform/kong-gw/release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: kong
spec:
  chart:
  ….
  ….
  values:
    image:
      repository: kong/kong-gateway
      tag: "3.1.1.3"
    replicaCount: 1
—--

Configuring Kong Environment Variables

Within the file, you may notice the section: spec.values.env. This section is one that allows the user to overwrite the default Kong Gateway configuration parameters via the use of environmental variables. This is a very important feature, which is documented here. An example of some of the environmental variables that you would wish to overwrite is shown below:



#platform/kong-gw/release.yaml
……
   env:
      prefix: /kong_prefix/
      database: postgres
      # Pre-created Kubernetes ExternalName that pointing to the PG Host
      pg_host: kong-db-postgresql.kong-enterprise.svc.cluster.local
      pg_port: 5432
      pg_user: konger
      pg_database: kong # Pre-create in RDS First
      pg_password: 
        valueFrom:
          secretKeyRef:
            name: kong-db-password # Pre-created previously
            key: postgresql-password # Pre-created previously

      # Logs Output
      log_level: warn

      # Configuring Admin GUI (Kong Manager) and Admin API
      admin_api_uri: http://admin.customer.kongtest.net:8001 
      admin_gui_url: http://manager.customer.kongtest.net:8002

      # Configuring Portal Settings 
      portal_gui_protocol: http
      portal_api_url: http://portalapi.customer.kongtest.net:8004
      portal_gui_host: portal.customer.kongtest.net:8003   
      portal_session_conf:
        valueFrom:
          secretKeyRef:
            name: kong-session-config
            key: portal_session_conf

      portal: on

Configuring Admin GUI (Kong Manager) and Admin API

Within the section, you'll find some important settings such as: admin_api_uri and admin_gui_url, where you have to indicate two hostnames that can be accessed by the API Operator using Kong Manager. admin_gui_url is the value Kong used to set in “Access-Control-Allow-Origin” header for CORS purposes, and it will decide which domain could access Kong Admin API via the web browser.

Configuring Portal Settings (Enterprise)

portal_gui_host is the hostname for your Developer Portal, portal_api_url and is the address on which your Kong Dev Portal API is accessible by Kong. Both settings are required if you are mapping your portal with your own DNS hostname.

portal_session_conf is the portal session configuration so as to create a session cookie that is to authenticate dev portal users. The sample snippet to create a portal_session_conf is found in the scripts/kong-gw-initial.sh

More information about portal session configuration can be found here: https://docs.konghq.com/gateway/latest/kong-enterprise/dev-portal/authentication/sessions/

Configuring Enterprise License and Default Kong Manager Secret

For an enterprise customer, one needs to create a license secret in the cluster by running the following:



kubectl create secret generic kong-enterprise-license --from-file=license=license.json -n kong --dry-run=client -o yaml | kubectl apply -f -

From the helm configuration, indicate the license_secret you just created, in this case the secret name is kong-enterprise-license.

For a better security posture, we enabled RBAC to secure our Kong Manager with basic-auth.To achieve this, we have to indicate the authentication mechanism via enterprise.admin_gui_auth, and enterprise.session_conf_secret for the default password.

You can read more about this here: https://docs.konghq.com/gateway/latest/kong-manager/auth/basic/



#platform/kong-gw/release.yaml
……
   enterprise:
      enabled: true
      # CHANGEME: https://github.com/Kong/charts/blob/main/charts/kong/README.md#kong-enterprise-license
      license_secret: kong-enterprise-license
      vitals:
        enabled: true
      portal:
        enabled: true
      rbac:
        enabled: true
        admin_gui_auth: basic-auth
        session_conf_secret: kong-session-config
        admin_gui_auth_conf_secret: kong-session-config
      smtp:
        enabled: false

Configuring Kong Services

As you might have known, a single Kong container comprised core services to make it work.

Kong Manager - Admin UI for you to manage API. Interact with Kong Admin API behind the scene
Kong Admin API - Admin API for you to manage API
Kong Developer Portal - Developer Portal to browse and search API documentation, and test API endpoints
Kong Developer Portal API - API to interact with Dev Portal.
Kong Proxy - Proxy service that process API request

To deploy these services, it is just as simple as enabling them in the helm values. For this exercise, we will only enable Kong Proxy and Admin API.



#platform/kong-gw/release.yaml
……
   admin:
      enabled: true
      type: LoadBalancer
      annotations: 
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb" 
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"
   manager:
      enabled: false
   proxy:
      # Creating a Kubernetes service for the proxy
      enabled: true
      type: LoadBalancer 
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb" 
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"
  portal:
      enabled: false

  portalapi:
      enabled: false

The full sample HelmRelease CRD can be accessed here : https://github.com/robincher/kong-flux-gitops/blob/main/platform/kong-gw/release.yaml

Now, we can update and check-in the above two configuration file to the following locations:

Bootstrapping our cluster with Flux + Kong

After the cluster has been created and configuration updated, we need to bootstrap the cluster with the Flux CLI. Bootstrapping is a Flux process that's used to install the Flux CRD's and then deploy a sample application, which in our case is Kong. To find out more about what bootstrapping means, see the Flux documentation for information regarding bootstrapping.

Before running the bootstrap command, we need to retrieve a Github Personal Access Token(PAT). Without a PAT, the Flux CLI will not be able to access Github API to do the necessary action like creating a fresh source repository if it doesn’t exist.

The command on how to bootstrap Kong is shown below:

Create an environment variable based on your Github PAT. This token will be used by Flux to interact with the corresponding Github repository



export GITHUB_TOKEN=<your-token>

Run the following command to bootstrap Flux into the remote EKS Cluster



flux bootstrap github \
 --owner=your-github-id \
 --repository=kong-flux-gitops \
 --path=clusters/staging \   #The Folder that contain the manifest
 --personal

Once you've bootstrapped the cluster, you should now have the Flux CRDs installed in your cluster and also see the cluster being initiated with Kong pods.

The flux bootstrap command perform the followings:

A new Git repository for our manifests is created on the source repository
A flux-system namespace with all Flux components is configured on our cluster



kubectl get pods -n flux-system

Expected output with all Flux’s components up and running



❯ 
NAME                                       READY   STATUS    RESTARTS   AGE
helm-controller-56fc8dd99d-sf4ml           1/1     Running   0          4d2h
kustomize-controller-bc455c688-mwfv4       1/1     Running   0          4d2h
notification-controller-644f548fb6-69wr4   1/1     Running   0          4d2h
source-controller-7f66565fb8-q4r7j         1/1     Running   0          4d2h

The Flux controllers are set up to sync with our new git repository



flux get sources git

Expected output with all Flux’s components up and running



NAME        REVISION        SUSPENDED   READY   MESSAGE
flux-system main/43187e2    False       True    stored artifact for revision 'main/43187e206a7d6f3e06406afc17b9579dec3ee04d'

After the sync is established, any Kubernetes manifests checked into the source repository will be automatically deployed into the target cluster.

Let’s check if all the Kong pods are up and running



kubectl get pods -n kong

Expected output



NAME                              READY   STATUS      RESTARTS   AGE
kong-kong-665fc7d8db-m67lb        1/1     Running     0          2m2s
kong-kong-init-migrations-62c7x   0/1     Completed   0          2m2s



kubectl get services -n kong

Expected output



NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP                            PORT(S)                         AGE
kong-kong-admin      LoadBalancer   10.100.0.98     xx2.elb.ap-southeast-1.amazonaws.com   8001:31386/TCP,8444:30909/TCP   2m21s
kong-kong-proxy      LoadBalancer   10.100.140.25   xx1.elb.ap-southeast-1.amazonaws.com   80:31675/TCP,443:30040/TCP      2m21s

Making Kong Config Updates

As shown above there is only 1 x Kong Pod, so what can we do to scale it up via GitOps?

Let’s update the replica count to 2



values:
    image:
      repository: kong/kong-gateway
      tag: "3.1.1.3"
    replicaCount: 2 # Change to this

Commit the code, and push to the remote repository.



git add kong-gw/release.yaml
git commit -m "chore: Bump 2 x Kong pod"

git push origin main

Observe the number of Kong Pods now by running the below command




❯ kubectl get pods -n kong-enterprise
NAME                                      READY   STATUS      RESTARTS   AGE
kong-kong-665fc7d8db-m67lb                1/1     Running     0          5m
kong-kong-665fc7d8db-nsqm7                1/1     Running     0          26s

You should now see there are 2 x Kong pods running, and what happened is due to Flux pulling the new manifest from the source repository it's synchronized with, and then updating the state of the cluster accordingly.

Cleaning-Up

Before you receive any bill shock, do remember to destroy the EKS cluster upon the conclusion of your experiment. Below are the steps to clean up and destroy the cluster upon experimentation

Uninstall Flux


 uninstall --namespace=flux-system

Clean up Kong Resource

Do a full-clean up of any remaining Kong resources


 delete ns kong

Delete the node groups

eksctl delete nodegroup --name Kong-GitOps-Test-Cluster --region <aws-region>

Delete the cluster



 delete cluster --name Kong-GitOps-Test-Cluster  --region <aws-region>

Summary

The key message here is that Kong is flexible and agnostic enough to be installed using several methods. With Gitops tooling like Flux, it can support all Kong Kubernetes deployment flavours with their operators or controllers, and have them consistently deployed across clusters.

With GitOps, you can easily automate configurations across multiple clusters. This is critical if ,for any unfortunate event, that you have to rebuild your current cluster in another site. With Flux GitOps, you can easily replicate the configuration by bootstrapping the new cluster with Flux and pointing to the existing source repository like what we did in this exercise. As a result, downtime is decreased and services are impacted minimally.

Kong supports the following deployment flavours for Kubernetes, and you can easily use them with Flux GitOps

Helm
Kubernetes Custom Resource Definition (CRD)
Kubernetes Operators (Incubator)

Sample Repository can be found here: https://github.com/robincher/kong-flux-gitops

Securing your site via OIDC, powered by Kong and KeyCloak

Robin Cher — Wed, 19 Jan 2022 06:30:33 +0000

Quick sharing on how you can further secure your api or endpoints with OIDC, and powered by Kong and Keycloak. The examples shared are all open-source solutions.

Introduction

As we start shipping services out to the open world, one of the primary considerations is how we can prevent any malicious attacks, since now we are in the wild west.Thankfully, We have the opportunity to tinker and experiment with open-source solutions due our work nature at that time. In this post, i would like to share how we incorporate Kong Ingress Controller, KeyCloak and Kubernetes to have an initial OIDC flow to front our external services.

System Context

Before we dive deeper, let's take a closer look on how OIDC will verify the authenticity of a user before allowing the request to be fulfilled.

OIDC (OpenID Connect) Overview

Extracted from : https://github.com/nokia/kong-oidc

Preparation

Before we begin, we need to have the following :

Kong - An API Gateway (community edition is open source and free)
Kong OIDC Plugin - Open-sources OIDC plugin for Kong, maintained by the community
Kong JWT KeyCloak Plugin - Plugin for Kong so as to allow JWT token to be issued and validated by Keycloak. Maintained by the community
Keycloak - A OpenID Connect Provider (Open Source)
Kubernetes - Open-source container orchestration system where we will deploy Kong and KeyCloak

Baking the OIDC Plugin with Kong Base Image

Important Note : This is a community maintained OIDC Plugin. If required, please consider checking out the official OIDC Plugin that is only available with Kong Enterprise

As we are using the open-source plugin by Revomatico, we need to bake the plugin with the base kong image.

Sample Dockerfile



FROM kong/kong:2.7.0

ENV OIDC_PLUGIN_VERSION=1.2.3-2
ENV JWT_PLUGIN_VERSION=1.1.0-1
ENV GIT_VERSION=2.24.4-r0
ENV UNZIP_VERSION=6.0-r7
ENV LUAROCKS_VERSION=2.4.4-r1


USER root
RUN apk update && apk add git=${GIT_VERSION} unzip=${UNZIP_VERSION} luarocks=${LUAROCKS_VERSION}
RUN luarocks install kong-oidc

RUN git clone --branch v1.2.3-2 https://github.com/revomatico/kong-oidc.git
WORKDIR /kong-oidc
RUN luarocks make

RUN luarocks pack kong-oidc ${OIDC_PLUGIN_VERSION} \
     && luarocks install kong-oidc-${OIDC_PLUGIN_VERSION}.all.rock

WORKDIR /
RUN git clone --branch 20200505-access-token-processing https://github.com/BGaunitz/kong-plugin-jwt-keycloak.git
WORKDIR /kong-plugin-jwt-keycloak
RUN luarocks make

RUN luarocks pack kong-plugin-jwt-keycloak ${JWT_PLUGIN_VERSION} \
     && luarocks install kong-plugin-jwt-keycloak-${JWT_PLUGIN_VERSION}.all.rock

USER kong

Build and push the image to a registry



# Build the container with any new changes
docker build -t <reponame>/kong-oidc:<tag> -f Dockerfile . 

# Run the container in detached mode
docker run -d --name kong-oidc <reponame>/kong-oidc:<tag>       

# Pushing the container image to a registry
docker push <reponame>/kong-oidc:<tag>

Understand the Kong Cluster Plugin Configurations

Kong Ingress Controller allow us to configure Kong specific features using several Custom Resource Definitions(CRDs)

These are the following CRDs that we need to take note of

odic - This plugin is used to communicate with the Keycloak Identity provider and is required if you'd like to enable (recommended) SSO for your ingress.
request-transformer - To strip off unnecessary headers upon authentication with the identity platform
cors - Allow Cross Site origin at global level

Deploying into Kubernetes

Examples manifest can be found on here

#### Deploy the previously baked Kong image (with OIDC plugin).



kubectl apply -f kong-deployment.yaml

#### Deploy KeyCloak



kubectl apply -f keycloak-deployment.yaml

KeyCloak Configurations

Create a new Realm in KeyCloak

Create a new Kong Client in the realm , eg kong-oidc

Client Configurations

Go to Clients, and then click on Settings.

Make the following updates

Access Type: Confidential
Valid Redirect URIs: *
Web Origin: localhost (Allowed CORS origin)

Copied the Client ID, and then go to Credentials to get the Secret value.

Retrieve OpenID Endpoint Configuration

Go to Realm Setting, and select General. Click on OpenID Endpoint Configuration

Override the values of the OIDC Kong Plugins



apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: oidc
  annotations:
    kubernetes.io/ingress.class: "kong"
  labels:
    global: "false"
disabled: false # optionally disable the plugin in Kong
plugin: oidc
config: # configuration for the plugin
  client_id: kong-oidc # Realm
  client_secret: xxxxxxxx  # Client Secret Copied
  realm: kong
  discovery: https://localhost/.well-known/openid-configuration # OpenID Endpoint Configuration Copied
  scope: openid
  redirect_after_logout_uri : https://localhost/auth/realms/kong/protocol/openid-connect/logout?redirect_uri=https://localhost

Deploy the required Cluster Plugins as mentioned above



kubectl apply -f kong-crds.yaml

Testing your site with OIDC

We can now test a sample ingress that is intercepted by Kong Ingress Controller. All you have to do is indicating the plugins annotations on the ingress



# sample-oidc.yaml
....
....
----
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echoserver-oidc
  namespace: default
  annotations:
    konghq.com/plugins: request-transformer,oidc #indicate here
spec:
  ingressClassName: kong
  rules:
    - host: echo-oidc.localhost.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: 
                name: echoserver
                port: 
                  number: 80

You should be prompted to sign in when attempting to access the site, and have successfully implement a minimal OIDC solution using open-source technology !

Learnings

We choose Kong because of its flexibility (we only scratch its surface on what it capable of as a full-fledge API Gateway) and being lightweight, which allow us to make changes quickly. Additionally, it is cloud-native which ease the effort of deployment.
KeyCloak is the de-facto IAM solution we have in our lab, and integrating it is straight forward enough. We don't really have to re-engineer much in the near future when we onboard more type of user pools (Social Logins, Azure Active Directory etc..)
If possible, start something small to see if it work as intended, especially when there are many inter-connected dots. It is always good to retain some technical acumen internally to allow better decision making when engaging with our software partners.

Credits

This work is not possible without the contribution from my following team mates:

Wee Liang - DevOps Engineer @ Thales Airlab
Arun - DevOps & Integration @ Thales Airlab
Kelvin - Software Engineer @ Thales Airlab

References

Shifting Left - Secret as Code with Flux V2 and Mozilla SOPS

Robin Cher — Mon, 10 Jan 2022 06:11:29 +0000

Introduction

A question that every engineers faced during their software development lifecycle is, "where should i stored my secrets?"

In this article, I will be sharing how my team "shift-left" to allow engineers self-service in managing their secrets. The guiding principle we adhering to is being as cloud-agnostic as possible.

Concepts

Before we dive deeper, let's align some concepts first. When we discuss about secret management for application, there are various way secret can be managed.

Writing them directly on source code as plain text. The anti-pattern and bad security practice, although it will work.
Putting the secrets somewhere else, like Cloud Provider solutions, Vault, File Systems in the Platform or Env Variables stored in the CI/CD.
Keeping them in the same code repository in an encrypted manner. Now we are talking about secrets lifecycle being managed via a pull request in a git-based workflow. This is the topic of secret as code we are talking about, where our Git Repository is the source of truth for secrets and configs.

Getting started

Before we start, we need to bootstrap flux to an existing cluster first.

Follow the instruction here if you need to: https://fluxcd.io/docs/get-started/

For my experiment, these are the technologies i use

Flux v2
Gitlab
Microsoft Azure Kubernetes Service
Microsoft Key Vault
Azure CLI
SOPs

Context

Below are the steps on how we can bootstrap an existing Kubernetes cluster to have the capability of decrypting SOPS secrets.

For my experimentation, i will be using Azure Key Vault for cryptographic operations. It is also supported by other Cloud Providers Key Management Service, or self generated PGP/GPG Key.

High-Level Setup

Here are the summarised steps to give Flux the capability in performing cryptographic operations. Flux's controller will be able to decrypt SOPs secret whenever the secret is being consumed by a pod.

Install Pod-Identity
Create Role Assignments for Kubelet
Create a managed identity
Create Azure KeyVault and Signing Key
Configure in-cluster secrets decryption

Detailed set-up :
https://techcommunity.microsoft.com/t5/azure-global/gitops-and-secret-management-with-aks-flux-cd-sops-and-azure-key/ba-p/2280068

Sample Repository:
https://github.com/robincher/bluesky-flux-sops-azure-template

Checking in Secrets

Before continuing , we need to install sops locally in our machine first.

Create a sample secret

# sample-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: demo-credentials
  namespace: default
type: Opaque
stringData:
  username: admin
  password: xxxxx

Encrypting the secret

Running the SOPs command with required parameters

sops -e --azure-kv <kv_key_address> --in-place --encrypted-regex "^(data|stringData)$" sample-secret.yaml > sample-secret-enc.yaml

Important flags to provide

a. azure-kv : Azure Keyvault Key Address, make sure you or the machine have the access

b. encrypyed-regex: Specifying what to be encrypted from the secret file.

Creating a default SOPS setting

Alternatively, we can create a default SOPS config that specify what to use for encryption, and where to be encrypted.

Create a .sops.yaml with the following values. We will encrypt all blocks under the stringData or data key.

# .sops.yaml
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    azure_keyvault: https://demo123.vault.azure.net/keys/demo-secret-key/c12231211

The default sops configs will be applied to its root and sub-folders.

Now we just have to run the following command to encrypt.

sops -e --in-place sample-secret.yaml > sample-secret-enc.yaml

You should get the following sample secret upon encryption

# sample-secret-enc.yaml
apiVersion: v1
kind: Secret
metadata:
    name: demoapp-credentials
    namespace: default
type: Opaque
stringData:
    username: ENC[AES256_GCM,data:21q4bo0=,iv:LOLxXQurjQR6cu9heQlZDdmhNgYO6VCBybbQHV6rO0w=,tag:58ep32CDrlCFuuDnD65VEQ==,type:str]
    password: ENC[AES256_GCM,data:oTZDkadQKL45dA==,iv:5VVbXC55xTVwH/n3t5gtKNtlkB3q7t8lW7Jw1czNSL0=,tag:WuqdubjTu6mQN5x1b3zDyw==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv:
        - vault_url: https://demo123.vault.azure.net
          name: demo-secret-key
          version: b7bc85c1a4ef4180be9d1de46725304c
          created_at: "2021-04-23T14:22:15Z"
          enc: KuFxRbcge198GU7hwHs078JNd_1EFtvcFqQ6bOLJDYMnWaW0kSbeD4DCxY0jX9MA17Rv3UMKHGfImgEbNfXGGIh7UucLPygpiuUyn9I73ClSQQ4trc4bD2yVkonCMwz5-0MiPVC3muhQpn3KjhThSucOgjhBnqQy_zzzzTeUP9PWi1pSp1jc3S2BxQIuKy09-oEakQogU4BRy55219befizYN7EFe8mstSIkvpksqGxKccH6dQum2k-OqsBUH2jkxiVgi5CEU35COy0pNWVJpZGuOaDMkGGqo7lrT4XKEGxtFKvEDxr6bTfjjQafuuxW9-4a9ZtaBkHCKopk66R9dcQ
    hc_vault: []
    age: []
    lastmodified: "2021-01-23T14:22:18Z"
    mac: ENC[AES256_GCM,data:as5mfREh5xdeiwbchkiiBS96tGuLJnEqme6VdDrPWKV9R0A4ATIM/1+HcbdAzGBXb8TmhO71hZMl3IvmX9DrNA/tvpPwFvLCkDfNhoWXJoXRRv6aRR7AJPlfcXkVMxxYaRDqz+ugAJkZG+5dhYeh1QAmiswjZOXaINEOw3Jf5dI=,iv:p/M2OhPdh2Naxu37Jt7EwiLf9Eb9OgExsmXX3hSUOJQ=,tag:fVqJ2jy++6GxHBPGXZHmHw==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.7.1

Testing

Let's check in the secret and see the magic by GitOps

rm sample-secret.yaml
git add sample-secret-enc.yaml
git commit -m "chore: Add encrypted secret"
git push

Check out the secret in Kubernetes

kubectl describe secret -n demoapp demoapp-credentials

Observation and Learning

We need to further educate the developers this new way of doing as the majority of them are used to checking in secrets into an external secret manager or stored as CI/CD variables.
Mozilla SOPs with Flux allow us to versioned our secrets as it follows a git-based workflow, and provide audit trails through the commit messages.
It can be deployed universally in both cloud or on-premises environment, as long as the cryptographic keys are backup and kept safely. We acknowledge that using Azure Key Vault might required additional work for us if we are to move to another provider's Key Management Service.
Using a cloud-based KMS alleviate the risk of having the decryption key being leaked (which typically being stored in the same machine), as there are access control in placed.
This is not a silver bullet to address all kind of secrets requirements as what work well for us might be going against the way of working for yours.

References

Moving to Pomerium Identity Aware Proxy

Robin Cher — Fri, 13 Nov 2020 11:03:37 +0000

Introduction

This is one of several learnings we encountered as part of Engineering Suite in Government Digital Services, GovTech SG. The focus for us in Engineering suite are creating productivity tools that supports our SG Tech Stack initiative.In this article, I will be sharing about how an Identity-Aware Proxy (IAP) enhanced my team’s security posture while making our developer's life happier

What is Identity Aware Proxy (IAP) ?

A proxy that enables secure access to upstream services. It can act as an identity attestation or perform delegated authorisation for these services.

It represents a paradigm shift from the traditional security model where the focus was securing the network perimeter through IP/Ports. With the proxy approach, not only are users verified, but the application requests can be terminated, examined, and authorised as well. IAP relies on application level access controls, whereby configured policies access user or application intent, not just ports and IPs.

Additionally, it supports the mantra of "Always verify, never trust" of a Zero-trust security model. You can read more about IAP as described by Google.

Image credit from Google IAP Overview

Our Scenario

Due to the Covid pandemic, most organizations have shifted to a remote working environment. It is by no means an easy feat to get everyone connected securely from their home, and the risk profile these days have changed. The last thing we want to do is to implement extra layers of security restrictions and constraints, resulting in a terrible developer's experience with marginal security gain. We wanted a solution that enables our users to access our workloads from untrusted networks with the use of yet another VPN. Traditionally, accessing the development toolchain required a different VPN service as the applications are hosted by a different team and it was impossible to leverage on their VPN service to front the team’s DEV environment.

We looked at a few IAP solutions, and eventually decided to proceed withPomerium mainly because it is open-sourced and our team have prior experience working with it. It is also very easy to deploy a full setup in a Kubernetes cluster

Decomposing Pomerium

Pomerium Proxy

It mainly interject and direct request to Authentication service so as to establish an identity from the idP. Additionally, it will process polices to determined internal/external route mapping.

Pomerium Authenticate (AuthN)

It handles authentication flow to the idP

Pomerium Authorize (AuthZ)

Processes policy to determine permissions for each service and handle authorization checks

Pomerium Cache

Stores session and identity data in persistent storage, and stores idP access and refresh tokens.

Implementation

The components required:

Kubernetes cluster with ingress
Azure Active Directory

In this article, i will be deploying Pomerium to a single Node in AWS EKS Cluster with NGINX ingress, and the perimeter are fronted by an Application Load Balancer. Azure Active Directory will be configured as the identity provider.

System Context

An overview on how pomerium are set-up, whereby we need to secure both an internal service that reside within the same cluster, and an external service that are hosted outside.

Additionally, will be listing down some of the good improvements that can be done in the later section.

Pomerium Configurations

Explaining Pomerium configuration in details.

Main Configurations

# Main configuration flags : https://www.pomerium.io/docs/reference/reference/
insecure_server: true
grpc_insecure: true
address: ":80"
grpc_address: ":80"

Leave this as default. This is to configure how Pomerium's services discovered each other. For this experiment , we can allow insecure traffic.

Workloads URL

## Workload URL
authenticate_service_url: https://authenticate.yourdomain.com

authorize_service_url: http://pomerium-authorize-service.default.svc.cluster.local

cache_service_url: http://pomerium-cache-service.default.svc.cluster.local

This is to define the route for required service. Pay special authentication to authenticate_service_url, which is a publicly discoverable URL that will be redirected to the web browser, while the other 2 are internal Kubernetes hostnames.

IDP Details

idp_provider: azure
idp_client_id: xxxxx
idp_client_secret: "xxx"
idp_provider_url : xxxx

Enter the details that you retrieve from the identity provider.

I will strongly recommend you not to hard code any secrets into the yaml files for production use. Alternative, you can create Kubernetes secrets and inject these as environmental variables.

For the list of supported identity provider and steps to generate the above details, please visit here

Policy

policy:
  - from: https://httpbin.yourdomain.com
    to: http://httpbin.default.svc.cluster.local:8000
    allowed_domains:
      - outlook.com

Policy contains route specific settings, and access control details. For this example, Pomerium will intercept all request to https://httpbin.example.com, and then redirect to the internal dns hostname of the httpbin workload.

Your final pomerium-config.yaml should look something like this

# Main configuration flags : https://www.pomerium.io/docs/reference/reference/
insecure_server: true
grpc_insecure: true
address: ":80"
grpc_address: ":80"

## Workload URL
authenticate_service_url: https://authenticate.yourdomain.com
authorize_service_url: http://pomerium-authorize-service.default.svc.cluster.local
cache_service_url: http://pomerium-cache-service.default.svc.cluster.local

idp_provider: azure
idp_client_id: REPLACE_ME.apps.googleusercontent.com
idp_client_secret: "REPLACE_ME"


policy:
  - from: https://httpbin.yourdomain.com
    to: http://httpbin.default.svc.cluster.local:8000
    allowed_domains:
      - outlook.com

Deploying

Pomerium configmap

We will create a configmap based on the configuration above, and then it will be mounted by the Pomerium workloads.

kubectl apply -f pomerium-config.yaml

Create random secret

kubectl create secret generic shared-secret --from-literal=shared-secret=$(head -c32 /dev/urandom | base64)
kubectl create secret generic cookie-secret --from-literal=cookie-secret=$(head -c32 /dev/urandom | base64)

Nginx controller

kubectl apply -f ingress-nginx.yaml

Mock Httpbin Services

Deploy a test internal httpbin service

kubectl apply -f httpbin-internal.yaml

Deploy Pomerium Workloads

kubectl apply -f pomerium-proxy.yml
kubectl apply -f pomerium-authenticate.yml
kubectl apply -f pomerium-authorize.yml
kubectl apply -f pomerium-cache.yml

Test

Ensure Pomerium workloads are up and running

Accessing Httpbin

From your web-browser, enter https://httpbin.yourdomain.com

You should be redirected to the IDP sign-in page.

So what happened here? Pomerium proxy interject your request based on the policy defined above, and then redirect to the identity provider's authentication page.

Enter your credentials, and you will be able to access Httpbin's page.

Additional use case

Promerium will be able to proxy services which reside outside of the EKS cluster by leveraging on EKS externalServiceName(insert link). By doing so, Services that reside outside of the EKS cluster will be able to have a service FQDN within EKS which will then be routable by Promerim.

Impact

Being able to protect our development and staging environments without another set of VPN greatly enhances the productivity of our engineering team. With Azure AD integration, we can carry out device posture checks, via conditional access policies, on the developer machine being used to access our protected environment. This improves the team’s security posture as a devices within a VPN boundary is no longer a guarantee of safety in today’s cybersecurity context.

Operationally, running the proxy on K8 facilitates capacity scaling in order to meet any sudden surge in demand. A properly instrumented K8 cluster can also help the DevOps team closely monitor all incoming network traffic.