Kong Hybrid Mode Deployment: GKE and On-Prem

Mastering Certificate Challenges with Ease

By Amrutha Paladugu
Author LinkedIn: https://www.linkedin.com/in/amrutha-paladugu/

Introduction:

Kong Hybrid mode represents a groundbreaking approach to API management, merging the control plane (CP) and data plane (DP) capabilities. This mode offers many benefits that empower organizations to achieve enhanced control, security, and performance in their API ecosystem. In this article, we’ll delve into a real-world scenario showcasing the capabilities of Kong Hybrid mode. Specifically, we’ll explore the deployment of Kong Hybrid across Google Kubernetes Engine (GKE) and an On-Premise Kubernetes cluster. We will focus on achieving secure communication, certificate management, workload deployment, and external access. Join us as we navigate this journey through a hybrid API management deployment.

Kong Hybrid architecture with one DP on the cloud and one DP on-premise.

Understanding Certificates:

Mutual Transport Layer Security (mTLS) plays a vital role in Kong’s security features by facilitating secure communication between services. When operating in hybrid mode, the authentication process relies on an mTLS handshake to ensure that communication between CP and DP nodes remains secure, requiring the presentation of certificates. Kong manages certificate key pairs in two distinct modes: Shared and PKI.

In the Shared mode (the default setting), Kong CLI generates a certificate/key pair, and copies of this pair are then distributed across nodes. CP and DP nodes share and utilize this certificate/key pair.

On the other hand, in the PKI mode, certificates signed by a central certificate authority (CA) are provided. Kong performs validation on both ends by verifying if they originate from the same CA. This approach effectively mitigates the risks associated with the transportation of private keys and provides increased security. If one of the DP nodes is compromised, an attacker won’t be able to affect other nodes in the Kong cluster.

In this article, we will use PKI mode and generate three different cert/key pairs for CP and the 2 DPs. All three pairs need to be signed by the same CA discussed above.
Additionally, we will provision a Let’s Encrypt certificate for DP1’s public-facing endpoint and verify encrypted traffic.

Certificate generation:

Step1: Generate CA cert using OpenSSL:

openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt

Step2: Generate private key and CSR(certificate signing request) for CP and DPs

openssl req -new -newkey rsa:2048 -nodes -keyout cp-tls.key -out cp-tls.csr -subj "/CN=*.kong.example.com"
openssl req -new -newkey rsa:2048 -nodes -keyout dp1-tls.key -out dp1-tls.csr -subj "/CN=*.kong.example.com"
openssl req -new -newkey rsa:2048 -nodes -keyout dp2-tls.key -out dp2-tls.csr -subj "/CN=*.127.0.0.1.nip.io"

The CN (Common Name) here would be your wildcard domain.

Step3: Use CA from Step1 to sign the generated CSRs from Step2

openssl x509 -req -in cp-tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out cp-tls.crt -days 365
openssl x509 -req -in dp1-tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out dp1-tls.crt -days 365
openssl x509 -req -in dp2-tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out dp2-tls.crt -days 365

Step4: Create bundle certs:

cat cp-tls.crt ca.crt > cp-tls-bundle.crt
cat dp1-tls.crt ca.crt > dp1-tls-bundle.crt
cat dp2-tls.crt ca.crt > dp2-tls-bundle.crt

Note: Save these certificates in an accessible directory (certs) for the CP and the DPs.

Install CP and DP1 on GKE:

Connect to a GKE cluster and use the next set of commands to install CP and DP1.

Create ns, Create secrets, update helm repo:

sudo snap install helm - classic
helm repo add kong https://charts.konghq.com
helm repo update
kubectl create namespace kong-cp
kubectl create namespace kong-dp1

Create secrets for Kong Enterprise license, Session Config, Admin password, and Postgresql password:

kubectl create secret generic kong-enterprise-license -n kong-cp \
 - from-file=license=license.json \
 - dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic kong-session-config -n kong-cp \
 - from-literal=portal_session_conf='{"storage":"kong","secret":"super_secret_salt_string","cookie_name":"portal_session","cookie_samesite":"off","cookie_secure":false, "cookie_domain": ".example.com"}' \
 - from-literal=admin_gui_session_conf='{"cookie_name":"admin_session","cookie_samesite":"off","secret":"super_secret_salt_string","cookie_secure":false,"storage":"kong", "cookie_domain": ".example.com"}'

kubectl create secret generic kong-manager-password -n kong-cp \
 - from-literal=password=password

kubectl create secret generic kong-cp-postgresql -n kong-cp \
 - from-literal=host="enterprise-postgresql.kong.svc.cluster.local" \
 - from-literal=password=password

Note: If you don’t have access to an Enterprise license, you can still go ahead and try the installation but will not be able to access the Admin GUI.

Now, create Kubernetes Secrets for the cluster cert and CA cert for ns kong-cp.

kubectl create secret tls kong-cluster-cert -n kong-cp \
 - cert=./certs/cp-tls-bundle.crt \
 - key=./certs/cp-tls.key

kubectl create secret generic kong-ca-cert -n kong-cp \
 - from-file=ca.crt=./certs/ca.crt

We use Helm to install the CP and DPs and use custom values.yaml files.

Update the values-cp.yaml with the below values:

image:
  repository: kong/kong-gateway
  tag: "3.4"

secretVolumes:
  - kong-cluster-cert
  - kong-ca-cert

env:
  role: control_plane
  cluster_mtls: pki
  cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt
  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  nginx_http_ssl_protocols: TLSv1.2
  admin_gui_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt
  admin_gui_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  admin_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt
  admin_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  portal_gui_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt
  portal_gui_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  portal_api_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt
  portal_api_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  lua_ssl_verify_depth: 3
  admin_api_uri: https://admin.kong.example.com
  admin_gui_url: https://manager.kong.example.com
  portal_auth: basic-auth
  portal_api_url: https://portal-api.kong.example.com
  portal_gui_url: https://portal-dev.kong.example.com
  portal_gui_host: portal-dev.kong.example.com
  portal_gui_protocol: https
  database: postgres
  pg_host: kong-cp-postgresql
  pg_database: kong
  pg_user: postgres
  pg_schema: kong
  pg_password:
    valueFrom:
      secretKeyRef:
        name: kong-cp-postgresql
        key: password
  portal_session_conf:
    valueFrom:
      secretKeyRef:
        name: kong-session-config
        key: portal_session_conf
  user: kong
  password:
    valueFrom:
      secretKeyRef:
        name: kong-manager-password
        key: password

admin:
  enabled: true
  type: ClusterIP
  annotations:
    konghq.com/protocol: https
  labels: {}
  http:
    enabled: false
  tls:
    enabled: true
    servicePort: 443
    containerPort: 8444
    parameters:
      - http2
  ingress:
    annotations:
      konghq.com/https-redirect-status-code: "301"
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    enabled: true
    ingressClassName: kong-dev
    hostname: admin.kong.example.com
    path: /
    tls: kong-cluster-cert

manager:
  enabled: true
  type: ClusterIP
  annotations:
    konghq.com/protocol: https
  labels: {}
  http:
    enabled: false
  tls:
    enabled: true
    servicePort: 443
    containerPort: 8445
    parameters:
      - http2
  ingress:
    annotations:
      konghq.com/https-redirect-status-code: "301"
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    enabled: true
    ingressClassName: kong-dev
    hostname: manager.kong.example.com
    path: /
    tls: kong-cluster-cert

cluster:
  enabled: true
  type: LoadBalancer
  annotations: {}
  http:
    enabled: false
  tls:
    enabled: true
    servicePort: 443
    containerPort: 8005
  ingress:
    annotations:
      konghq.com/https-redirect-status-code: "301"
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    enabled: true
    ingressClassName: kong-dev
    hostname: cp.kong.example.com
    path: /
    tls: kong-cluster-cert

clustertelemetry:
  enabled: true
  type: LoadBalancer
  http:
    enabled: false
  tls:
    enabled: true
    servicePort: 443
    containerPort: 8006
  ingress:
    annotations:
      konghq.com/https-redirect-status-code: "301"
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    enabled: true
    ingressClassName: kong-dev
    hostname: tel.kong.example.com
    path: /
    tls: kong-cluster-cert

proxy:
  enabled: false

ingressController:
  enabled: true
  ingressClass: kong-dev
  resources:
    limits:
      cpu: 200m
      memory: 256Mi
    requests:
      cpu: 100m
      memory: 128Mi
  installCRDs: false
  env:
    publish_service: kong-dp/kong-dp-kong-proxy
    kong_admin_token:
      valueFrom:
        secretKeyRef:
          name: kong-manager-password
          key: password

enterprise:
  enabled: true
  license_secret: kong-enterprise-license # CHANGEME
  vitals:
    enabled: false
  rbac:
    enabled: true
    session_conf_secret: kong-session-config
    admin_api_auth: basic-auth
  portal:
    enabled: true
  smtp:
    enabled: false

portal:
  enabled: true
  type: ClusterIP
  annotations: 
    konghq.com/protocol: https
  labels: {}
  http:
    enabled: false
  tls:
    enabled: true
    servicePort: 443
    containerPort: 8446
  ingress:
    annotations:
      konghq.com/https-redirect-status-code: "301"
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    enabled: true
    ingressClassName: kong-dev
    hostname: portal-dev.kong.example.com
    path: /
    tls: kong-cluster-cert

portalapi:
  enabled: true
  type: ClusterIP
  annotations: 
    konghq.com/protocol: https
  labels: {}
  http:
    enabled: false
  tls:
    enabled: true
    servicePort: 443
    containerPort: 8447
  ingress:
    annotations:
      konghq.com/https-redirect-status-code: "301"
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    enabled: true
    ingressClassName: kong-dev
    hostname: portal-api.kong.example.com
    path: /
    tls: kong-cluster-cert

postgresql:
  enabled: true
  auth:
    database: kong
    username: postgres
    existingSecret: kong-cp-postgresql
    secretKeys:
      adminPasswordKey: password

Let’s try and understand the above values file:

The secrets we created, kong-cluster-cert and kong-ca-cert are mounted as volumes for the SSL certificates and are accessed at the specified path, for example:

cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt

The environment variables are for configuring Kong and its components. They control settings related to SSL/TLS, database connections, authentication, and more. Also, they specify the role as “control_plane” and enable cluster mutual TLS (mTLS) using the PKI method. The admin API and GUI URLs, authentication methods, and other configuration parameters are also defined using the env variables.

Admin, Manager, Portal, Portal API, and PostgreSQL:

• These sections configure various components of Kong. All these components are enabled for the CP and would be disabled in the DP values yaml. Their type ( ClusterIP), TLS, and Ingress settings are also defined, including annotations, hostnames, paths, and TLS certificates for these components.
• PostgreSQL settings specify authentication details for connecting to the PostgreSQL database used by Kong.
• Enterprise-related settings include the license secret, RBAC configuration, SMTP, and vitals.

Proxy and Ingress Controller:

• The proxy section specifies whether the Kong proxy is enabled. For CP values, this would be disabled as the DP does all the proxying.
• Ingress Controller settings define the resources and environment variables for the Kong Ingress Controller.

Cluster and Cluster Telemetry:

• Cluster and cluster telemetry are essential for Kong Gateway’s Control Plane (CP) and Data Plane (DP) connection. DP instances connect to the CP through the CP cluster, which provides redundancy, load balancing, and configuration synchronization.
• Cluster telemetry plays a crucial role in monitoring the health of both CP and DP components, ensuring they can work together effectively and respond to changes and issues in real time. This setup provides a reliable and resilient CP-DP connection in Kong Gateway deployments.

Install the CP:

helm upgrade - install kong-cp kong/kong - namespace kong-cp -f values-cp.yaml

Once the CP is installed, verify that all pods are running and the Cluster and Clusterterlemetry services’ external LoadBalancer IPs are ready.

To install DP in the same cluster, follow the steps below and create license and certificate secrets for DP1.

kubectl create secret generic kong-enterprise-license -n kong-dp1 \
  --from-file=license=license.json \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret tls kong-cluster-cert -n kong-dp1 \
 - cert=./certs/dp1-tls-bundle.crt \
 - key=./certs/dp1-tls.key

kubectl create secret generic kong-ca-cert -n kong-dp1 \
 - from-file=ca.crt=./certs/ca.crt

To encrypt the oneway traffic to the public-facing end point of DP1, we use Let’s Encrypt CA. Use the below link to generate one for the domain you use.

SSL Certificate Generator: Free letsencrypt SSL in minutes:

A free SSL Certificate Generator. No login is required. Secure your site with a letsencrypt certificate. Includes a…
punchsalad.com

Create a secret for this cert in the kong-dp1 namespace:

kubectl create secret tls kong-tls-cert -n kong-dp1 \
 - cert=./letsencrypt/ca.crt \
 - key=./letsencrypt/ca.key
Create a values-dp1.yaml file with the below values:
image:
  repository: kong/kong-gateway
  tag: "3.4"

secretVolumes:
  - kong-cluster-cert
  - kong-ca-cert
  - kong-tls-cert

env:
  role: data_plane
  database: "off"
  status_listen: 0.0.0.0:8100
  cluster_control_plane: cp.kong.example.com:443
  cluster_telemetry_endpoint: tel.kong.example.com:443
  cluster_mtls: pki
  cluster_server_name: cp.kong.example.com
  cluster_telemetry_server_name: tel.kong.example.com
  lua_ssl_verify_depth: 3
  cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt
  lua_ssl_trusted_certificate: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  ssl_cert: /etc/secrets/kong-tls-cert/tls.crt
  ssl_cert_key: /etc/secrets/kong-tls-cert/tls.key

ingressController:
  enabled: false

enterprise:
  enabled: true
  license_secret: kong-enterprise-license
  vitals:
    enabled: false
  portal:
    enabled: false
  rbac:
    enabled: false

admin:
  enabled: false

manager:
  enabled: false

portal:
  enabled: false

portalapi:
  enabled: false

proxy:
  enabled: true
  type: LoadBalancer
  annotations: {}
  tls:
    enabled: true
    servicePort: 443
    containerPort: 8443
    parameters:
      - http2
  # stream: {}
  ingress:
    enabled: true

Please take note of the key differences here from the earlier values-cp yaml. All the Manager, Admin, Portal, Portalapi, and IngressController sections are disabled, while the Proxy section is enabled as type LoadBalancer. As explained earlier, ensuring the correct configuration of cluster and cluster telemetry endpoints is crucial.

Another point worth mentioning is the secret volume for the Let’s Encrypt certificate:kong-tls-cert. This is accessed through the below variables:

ssl_cert: /etc/secrets/kong-tls-cert/tls.crt
ssl_cert_key: /etc/secrets/kong-tls-cert/tls.key

Install DP1:

helm upgrade --install kong-dp1 kong/kong --namespace kong-dp1 -f values-dp1.yaml

Note the LoadBalancer IPs for the Cluster and Clustertelemetry service in the CP and the Proxy service in the DP. Map these values to the domain you intend to use. In our scenario, it would be as follows:

cp.kong.example.com - - - - - - - > 34.136.32.39

cp-tel.kong.example.com - - - - >34.68.58.255

*.kong.example.com - - - - - - - - >34.132.64.159

You should now be able to access Kong Manager (in case you are using an Enterprise license) at manager.kong.example.com, but the browser won’t recognize the OpenSSL CA, and hence it will appear as NotSecure as shown in the below image.

Access Admi GUI and view certificate (Not Secure- openSSL)

Deploy a sample application:

Let’s deploy a sample nodeJS application in the same GKE cluster in the default namespace, try accessing it through the DP endpoint, and see if the Let’s Encrypt certificate is served.

kubectl create deploy --image debianmaster/nodejs-welcome welcome 
kubectl expose deployment welcome --type=LoadBalancer --name=welcome --port=8080 --target-port=8080

Deploying a sample nodeJS ‘welcome’ application

Add a service and a route to this application through the Kong manager.

Now, the workload can be accessed through dp1.kong.example.com/welcome. Click on the lock icon🔒present right before the URL and view the certificate. You will notice that the Let’s Encrypt certificate is being served, as shown in the below image.

Access the workload through the DP endpoint and verify the Let’s Encrypt cert.

Install DP2 on local K3s:\

Connect to a local K3s (lightweight Kubernetes) cluster and install the following command set to install DP2.
Create namespace for DP2

kubectl create ns kong-dp2

Create secrets for enterprise licenses and certificates:

kubectl create secret generic kong-enterprise-license - from-file=license=./license.json -n kong-dp2 - dry-run=client -o yaml | kubectl apply -f -

kubectl create secret tls kong-cluster-cert -n kong-dp2 \
 - cert=./certs/dp2-tls-bundle.crt \
 - key=./certs/dp2-tls.key

kubectl create secret generic kong-ca-cert -n kong-dp2 \
 - from-file=ca.crt=./certs/ca.crt

Create a values-dp2.yaml file with the below values:

image:
  repository: kong/kong-gateway
  tag: "3.4"

secretVolumes:
  - kong-cluster-cert
  - kong-ca-cert

env:
  role: data_plane
  cluster_control_plane: cp.kong.example.com:443
  cluster_telemetry_endpoint: tel.kong.example.com:443
  lua_ssl_trusted_certificate: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_mtls: pki
  cluster_server_name: cp.kong.example.com
  cluster_telemetry_server_name: tel.kong.example.com
  cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt
  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  database: "off"
  audit_log: "on"
  headers: "off" 
  anonymous_reports: "off"
  untrusted_lua: "off"

enterprise:
  enabled: true
  license_secret: kong-enterprise-license
  vitals:
    enabled: false
  portal:
    enabled: false
  rbac:
    enabled: false
admin:
  enabled: false
manager:
  enabled: false
portal:
  enabled: false
portalapi:
  enabled: false
ingressController:
  enabled: false

proxy:
  enabled: true
  type: LoadBalancer
  annotations: {
    konghq.com/protocol: "https"
  }
  tls:
    enabled: true
    servicePort: 443
    containerPort: 8443
    parameters:
    - http2
  ingress:
    enabled: true
Install DP2
helm upgrade - install kong-dp2 kong/kong - namespace kong-dp2 -f values-dp2.yaml

Deploy a sample application in the K3s cluster:

Let’s deploy a sample nodeJS application in the same K3s cluster in the default namespace and try to access it through the DP2 endpoint and verify.

kubectl create deploy --image debianmaster/nodejs-welcome hello 
kubectl expose deployment hello --type=LoadBalancer --name=hello --port=8080 --target-port=8080

Verify deployment of nodeJS ‘hello’ application.

Add a service and a route to this application through the Kong manager.

Now the workload can be accessed through dp2.127.0.0.1.nip.io/hello

Conclusion:

Throughout this article, we embarked on a journey to uncover the transformative power of Kong Hybrid mode within API management. By harmonizing the control plane (CP) and data plane (DP), Kong Hybrid empowers organizations to balance control, security, and performance seamlessly. Through a real-world scenario, we showcased the deployment of Kong Hybrid across Google Kubernetes Engine (GKE) and an On-Premise Kubernetes cluster.

We navigated the intricate landscape of certificate management, emphasizing the pivotal role of Mutual Transport Layer Security (mTLS) and certificates in fortifying Kong’s security architecture. By embracing PKI mode, we ensured secure communication and mitigated the risks inherent in key transportation.

By following our straightforward, step-by-step instructions, we achieved successful deployments of Kong CP and DP1 within the GKE cluster, followed by Kong DP2 in an On-Premise K3s cluster. We highlighted the deployment of sample applications, demonstrated access through DP endpoints, and verified Let’s Encrypt certificates, reinforcing the foundation of a robust and secure API ecosystem.

As we wrap up this journey, remember the importance of resource cleanup to prevent lingering costs. The world of Kong Hybrid mode introduces a revolutionary approach that bridges the gap between Cloud and On-Premise, enabling the creation of APIs that transcend traditional confines. With mastery of this approach, you’ll confidently and innovatively navigate the dynamic realm of modern API management.

For more details visit: [https://zelarsoft.com/kong-consulting-and-licensing/]