Local Kubernetes Cluster - External traffic without Ingress Using Kftray

Hello. In this post, I'll show you how to set up a local Kubernetes cluster using Kind, Terraform, and Kftray. We'll keep all services inside the cluster, avoiding the need for ingress controllers or exposing services like NodePort or LoadBalancer. I'll also walk you through the Terraform code used in this setup.

Terraform code used in this Blog Post: https://github.com/hcavarsan/kftray-k8s-tf-example

Why Keep Services Inside the Cluster?

Exposing services externally can add complexity and potential security risks. For local development or secure environments, it's often better to keep everything internal. By using kubectl port-forward and automating it with Kftray, we can access services running inside the cluster without exposing them to the outside world.

Tools You'll Need

Before starting, make sure you have the following installed:

• Docker
• Terraform (v1.9.5)
• Kftray (you can choose between the GUI or TUI version)

Cloning the Repository

First, clone the repository that contains the Terraform code:

git clone https://github.com/hcavarsan/kftray-k8s-tf-example
cd kftray-k8s-tf-example/terraform

Understanding the Terraform Code

The Terraform code in this repository automates the following:

• Creates a Kind Kubernetes cluster.
• Deploys Helm charts for Argo CD, Prometheus, Alertmanager, Grafana, and Jaeger.
• Sets up service annotations for Kftray to auto-import port-forward configurations.

Project Structure

Here's how the project is organized:

kftray-k8s-tf-example
├── terraform
│   ├── helm.tf
│   ├── outputs.tf
│   ├── locals.tf
│   ├── providers.tf
│   ├── variables.tf
│   ├── templates
│   │   ├── argocd-values.yaml.tpl
│   │   ├── grafana-values.yaml.tpl
│   │   ├── jaeger-values.yaml.tpl
│   │   └── prometheus-values.yaml.tpl
│   └── kind.tf
├── Makefile
├── docs
│   ├── kftray.gif
│   └── kftui.gif
└── README.md

providers.tf

This file specifies the Terraform providers we'll use:

terraform {
  required_version = ">= 1.0.0"

  required_providers {
    kind = {
      source  = "tehcyx/kind"
      version = "0.4.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.0.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.0.0"
    }
    template = {
      source  = "hashicorp/template"
      version = ">= 2.0.0"
    }
  }
}

provider "kind" {
}

provider "kubernetes" {
  config_path = kind_cluster.default.kubeconfig_path
}

provider "helm" {
  kubernetes {
    config_path = kind_cluster.default.kubeconfig_path
  }
}

• kind: Manages Kind clusters.
• kubernetes: Interacts with the Kubernetes cluster.
• helm: Deploys Helm charts.
• template: Processes template files.

variables.tf

Defines variables used in the Terraform configuration:

variable "cluster_name" {
  description = "Name of the Kind cluster"
  type        = string
  default     = "kftray-cluster"
}

variable "kubernetes_version" {
  description = "Version of the Kind node image"
  type        = string
  default     = "v1.30.4"
}

variable "kubeconfig_dir" {
  description = "Directory to store the kubeconfig file"
  type        = string
  default     = "~/.kube"
}

# Chart versions
variable "argocd_chart_version" {
  description = "Version of the Argo CD Helm chart"
  type        = string
  default     = "5.19.12"
}

variable "prometheus_chart_version" {
  description = "The version of the Prometheus chart to deploy."
  type        = string
  default     = "25.27.0"
}

variable "grafana_chart_version" {
  description = "The version of the Grafana chart to deploy."
  type        = string
  default     = "8.5.0"
}

variable "jaeger_chart_version" {
  description = "The version of the Jaeger chart to deploy."
  type        = string
  default     = "3.3.1"
}

kind.tf

Creates the Kind cluster:

resource "kind_cluster" "default" {
  name            = var.cluster_name
  node_image      = "kindest/node:${var.kubernetes_version}"
  kubeconfig_path = pathexpand("${var.kubeconfig_dir}/kind-config-${var.cluster_name}")
  wait_for_ready  = true

  kind_config {
    kind        = "Cluster"
    api_version = "kind.x-k8s.io/v1alpha4"

    node {
      role = "control-plane"

      extra_port_mappings {
        container_port = 80
        host_port      = 80
        protocol       = "TCP"
      }
    }

    node {
      role = "worker"
    }
  }
}

• name: Cluster name.
• node_image: Kubernetes version.
• kubeconfig_path: Where to store the kubeconfig file.
• wait_for_ready: Wait until the cluster is ready.
• kind_config: Custom Kind configuration.

locals.tf

Defines local variables and service configurations:

locals {
  services = {
    argocd = {
      namespace  = "argocd"
      repository = "https://argoproj.github.io/argo-helm"
      chart      = "argo-cd"
      version    = var.argocd_chart_version
      kftray = {
        server = {
          alias       = "argocd"
          local_port  = "16080"
          target_port = "http"
        }
      }
    }
    # ... other services ...
  }

  services_values = {
    for service_name, service in local.services :
    service_name => templatefile("${path.module}/templates/${service_name}-values.yaml.tpl", {
      kftray = service.kftray
    })
  }
}

• services: A map of services to deploy.
• kftray: Port-forward configurations for Kftray.
• services_values: Processes Helm values templates for each service.

helm.tf

Deploys the services using Helm:

resource "helm_release" "services" {
  depends_on = [kind_cluster.default]
  for_each         = local.services
  name             = each.key
  namespace        = each.value.namespace
  create_namespace = true
  repository       = each.value.repository
  chart            = each.value.chart
  version          = each.value.version

  values = [
    local.services_values[each.key]
  ]
}

• for_each: Iterates over each service.
• name: Release name.
• namespace: Kubernetes namespace.
• repository: Helm chart repository.
• chart: Helm chart name.
• version: Chart version.
• values: Custom values for the Helm chart.

templates/

Contains Helm values templates for each service (e.g., argocd-values.yaml.tpl). These templates inject the Kftray annotations into the service definitions.

outputs.tf

Defines outputs for the Terraform run:

output "endpoint" {
  description = "API endpoint for the Kind cluster."
  value       = kind_cluster.default.endpoint
}

output "kubeconfig" {
  description = "Kubeconfig file for the Kind cluster."
  value       = kind_cluster.default.kubeconfig
  sensitive   = true
}

output "credentials" {
  description = "Credentials for authenticating with the Kind cluster."
  value = {
    client_certificate     = kind_cluster.default.client_certificate
    client_key             = kind_cluster.default.client_key
    cluster_ca_certificate = kind_cluster.default.cluster_ca_certificate
  }
  sensitive = true
}

Applying the Terraform Configuration

To apply the Terraform configuration and set up the cluster, run:

make apply

This will:

• Initialize Terraform.
• Create the Kind cluster.
• Deploy the Helm charts.
• Set up the service annotations.

Installing Kftray

Go to the Kftray GitHub page and follow the installation instructions for your operating system.

Importing Port-Forward Configurations into Kftray

Using Kftray GUI

1. Open Kftray and click the tray icon to open the main window.
2. Click the menu icon at the bottom left corner.
3. Select "Auto Import."
4. Click "Set kubeconfig" and choose the kubeconfig file created by Terraform (usually ~/.kube/kind-config-kftray-cluster).
5. Select the context kftray-cluster from the dropdown menu.
6. Click "Import" to load the port-forward settings.

After importing, you can start port forwarding by toggling the switch next to each service or by clicking "Start All."

Using Kftui (Terminal Interface)

1. Set the KUBECONFIG environment variable:

   export KUBECONFIG="~/.kube/kind-config-kftray-cluster"

1. Start Kftui:

   kftui

1. Press Tab to access the top menu and select "Auto Import."
2. Press Ctrl+A to select all configurations.
3. Press F to start all port forwards.

Accessing Your Services Locally

With port forwarding set up, you can access your services on your local machine:

• Argo CD: http://localhost:16080
• Prometheus: http://localhost:19090
• Alertmanager: http://localhost:19093
• Grafana: http://localhost:13080
• Jaeger: http://localhost:15090

Custom Kftray Settings

Adjusting Kftray Port Forwarding Settings

To customize how Kftray forwards ports, edit the locals.tf file:

locals {
  services = {
    argocd = {
      kftray = {
        server = {
          alias       = "argocd"
          local_port  = "16080"
          target_port = "http"
        }
      }
    }
    # Other services...
  }
}

• alias: The name displayed in Kftray.
• local_port: The port on your machine to access the service.
• target_port: The service's port or port name inside the cluster.

Cleaning Up

If you want to destroy the cluster and remove all resources, run:

make destroy

This will tear down the cluster and delete all the resources created by Terraform.

Conclusion

By keeping all services inside the cluster and using Kftray for port forwarding, we create a simpler and more secure environment. This setup is useful for local development and situations where you want to avoid exposing services externally.

Feel free to explore and modify the Terraform code to suit your needs. If you have any questions or need help, feel free to reach out.

Thanks for reading. You can find more of my work or get in touch on GitHub.