DEV Community: Alex Robson

Local Kubernetes - Adding An Ingress Controller

Alex Robson — Tue, 17 Jan 2023 09:25:31 +0000

Previous Posts in This Series

A Brief Explanation of Kubernetes Networking

Unlike running Docker containers with ports bound to a host port, Kubernetes does not expose container ports or assign them an IP address. Kubernetes has a service resource that exposes ports in a POD to a named endpoint and port.

A service provides a predictable way to access containers via the internal cluster network. The container will only be reachable from within the cluster through the service.

In the last post, we used m8s dashboard-proxy to make the kubernetes-dashboard service accessible outside the cluster.

To see the manifest of the kubernetes-dashboard service, issue the following:

k8s get service -n kube-system kubernetes-dashboard -o yaml

Note: The service binds the port for this named service to the targetPort on the Pod.

To display the manifest for the dashboard pod, issue the command:

k8s describe pod -n kube-system kubernetes-dashboard

Look for the ports collection inside the container spec for kubernetes-dashboard to see the port setting 8443/TCP.

Enabling DNS

Kubernetes' DNS provides service discovery, a valuable feature when containers can disappear or get added, resulting in a shifting set of IP addresses.

To add DNS, type:

m8s enable dns

Editing The Dashboard's Deployment and Service

For safety, the dashboard is secure by default and allows HTTPS only through a certificate it creates. I want to demonstrate how to host the dashboard through the ingress controller through port 80. In a future post, we'll secure this with SSL termination on the ingress controller.

The dashboard will require changes to host HTTP traffic and updating the service to bind to a different target port.

Changing The Deployment

To fetch the current deployment manifest for the dashboard, use the following:

k8s get deployment -n kube-system kubernetes-dashboard -o yaml > dashboard-deployment.yml

Open your favorite editor and follow each set of directions.

Enable Insecure Login

Change --auto-generate-certificates to --enable-insecure-login to forgo generating self-signed certificates and bind the dashboard process to port 9090.

Change 8443 to 9090

We need to change the port the container will expose to 9090 from 8443 under the ports section.

Update the Liveness Probe

Edit the liveness probe section by changing the httpGet's port to 9090 and the scheme to HTTP.

Apply the Changes

Save the changes you made to the file and apply our changes to the deployment; issue the following:

k8s apply -f ./dashboard-deployment.yml

Changing The Service

Now that our dashboard deployment is bound to port 9090, we need to update our service. To fetch the service manifest, type:

k8s get service -n kube-system kubernetes-dashboard -o yaml > dashboard-service.yml

Change The Ports

We'll change the port from 443 to 80 and targetPort from 8443 to 9090.

Apply the Changes

Save your changes and update the service as we did with the deployment with the following:

k8s apply -f ./dashboard-service.yml

Adding the NGINX Ingress Controller

An Ingress Controller is how Kubernetes accepts incoming HTTP requests through a fixed set of IPs and directs them to the correct backing Pods/Containers based on configuration.

To install the NGINX Ingress Controller from Kubernetes, we can enable the add-on as follows:

microk8s enable ingress

Creating Ingress for The Dashboard

Save the following YAML for the manifest in a file http-dashboard-ingress.yml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: http-ingress
  namespace: kube-system
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2    
spec:
  rules:
  - http:
      paths:
      - path: /dash(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 80

From your shell, the following command will cause the ingress controller to update the NGINX configuration.

k8s create -f ./http-dashboard-ingress.yml

The new NGINX configuration forwards all requests to port 80 to the dashboard container.

Finding The Cluster IP

To access the dashboard, we'll need the IP address of the cluster. The endpoint slice named kubernetes should resolve to the Ingress controller. Fetch the list of endpoint slices with the following:

k8s get endpointslices

Accessing The Dashboard

The dashboard should now be accessible via port 80 at the URL /dash/. The dashboard should display a notification in red at the bottom notifying you that authentication is disabled since you're accessing it via an unsecured (HTTP) connection through an IP other than localhost or 127.0.0.1.

Creating a Proxy

The proxy built into microk8s is not compatible with the changes made to the dashboard deployment. The port-forward feature in Kubernetes creates a tunnel from a specified resource to the localhost.

k8s -n ingress port-forward service/ingress 8008:80

Now the dashboard can be accessed http://localhost:8008/dash/ and will accept authentication.

Up Next

In the next post, I'll look at options for SSL termination.

References

Kubernetes Services

Microk8s Addon: Ingress

NGINX Ingress Controller

Local Kubernetes RBAC & Dashboard Setup

Alex Robson — Tue, 17 Jan 2023 04:00:28 +0000

In my previous post, I showed how anyone could install a Kubernetes cluster locally using Canonical's microk8s.

This post picks up where the last ended. Before I start looking at deploying software to my local cluster, I need to do some additional configuration.

Note: I've aliased microk8s as m8s and microk8s kubectl as k8s.

Kubernetes Dashboard

The Kubernetes Dashboard is a web interface that makes discovery and simple observability straightforward. microk8s provides a dashboard package that you can install with:

m8s enable dashboard

To view the dashboard, use the dashboard proxy command:

m8s dashboard-proxy

To open the dashboard, ctrl+click the URL in the console. Bypass your browser's complaint that the certificate is invalid*, copy the token at the end of the output, and use that to authenticate when prompted in the browser.

Role-Based Authentication Control (RBAC)

What is it?

Though Kubernetes is usable without it, RBAC allows you to practice good security hygiene and work in an environment closer to production clusters.

If you're not familiar, RBAC consists of the following:

service accounts
roles
role bindings

Since you can bind multiple roles to a service account, this is a very flexible way to add or remove sets of permissions to an account.

Enabling RBAC

Enabling RBAC is simple:

microk8s enable rbac

Fixing The Dashboard Account's Permissions

Installing RBAC and the dashboard will create a service account for kubernetes-dashboard. To see a list of service accounts that includes this new account, issue:

k8s get sa -n kube-system

To see the other RBAC resources created for us to make the dashboard accessible, issue the following:

Lists Cluster Roles

k8s get clusterroles -n kube-system | grep dashboard

Lists Role Bindings

k8s get clusterrolebindings -n kube-system | grep dashboard

Unfortunately, the permissions granted to the kubernetes-dashboard account through the cluster role kubernetes-dashboard are so restrictive that we won't get much visibility into what's already there.

Rather than dive into the specifics of each manifest, I've put a new clusterrole, clusterrole-binding, and token file into a gist you can use to:

Create a new cluster role named kubernetes-dashboard-readonly
Create a cluster role binding named kubernetes-dashboard-readonly
Create an authentication token for the kubernetes-dashboard account

Save the contents of the gist to ./read-only-dashboard.yml and then run the command:

k8s create -f ./read-only-dashboard.yml

To retrieve the token for login with the kubernetes-dashboard use:

k8s describe secrets -n kube-system kubernetes-dashboard-token

Copy the value following token and use this to log into the dashboard. The first sign of success is that the namespaces drop-down in the top bar has more than default.

Help With Custom Roles

Finding a list of the available API groups and their related resources with their allowed verbs has been a headache. There is a built-in command that will provide a tab-delimited table:

k8s api-resources -o wide

To make this easier to consume (for myself), I put this in a Google Sheet and sorted it by the API Group and Resource Name columns. You can view it here.

Up Next

In my next post, I plan to add the Kubernetes Ingress NGINX Controller and demonstrate how this creates a reverse proxy and load balancer for services we may want to expose outside the cluster.

*Known Issues With Self-Signed Certificate

Following these directions on a machine that does not allow you to proceed to an HTTPS URL with a self-signed certificate will prevent you from viewing the dashboard.

In the next post, I'll look at ways to get a valid certificate for an ingress controller.

Resources

Gist: Read-only Dashboard Role

Kubernetes API Groups and Resources

Installing A Local Kubernetes

Alex Robson — Mon, 16 Jan 2023 01:18:35 +0000

Why

There are good reasons to follow along and get a local Kubernetes cluster:

learn more about containers and orchestration
experimentation is a great way to learn
gain new skills and understanding

I will be writing more about CI/CD tooling that targets Kubernetes, and starting with documenting how I created my local cluster makes what I'm writing about easier for others to reproduce.

What's Kubernetes?

Kubernetes is an Open Source container orchestration system. Kubernetes' API provides a uniform control layer for managing containerized services. There is a lot of great material available that explains what Kubernetes is. See Further Reading at the end for some suggestions.

After five years managing physical servers, then another four years working with VM clusters, the value of Linux Containers(LXC) and their eventual productization as Docker appealed to me.

Three years of creating and evolving internal tooling to manage hosting and configuration of Dockerized microservices ended when I discovered Kubernetes in the fall of 2016.

At that time, I migrated a pricy Heroku deployment to a self-managed Kubernetes cluster in AWS that expanded our capabilities, reduced maintenance efforts, increased capacity, and slashed our hosting costs by more than half.

How

Today there are quite a few ways to get a Kubernetes experience on a local machine. Here are a few that I'm most familiar with and have tried at one point or another:

microk8s

microk8s is Canonical's packaged Kubernetes cluster purpose-built for more straightforward installs. Like Rancher Lab's k3s, Canonical built microk8s for local development, edge computing, and IoT applications.

My Environment

For this post, I chose my Windows desktop running the following:

Windows 10.0.19044.2364
WSL2 1.0.3.0
Debian GNU/Linux 11 (bullseye)
Kernel Linux 5.15.79.1-microsoft-standard-WSL2

Some of the steps I share are specific to my environment. You may have to adjust the approach to work for you if you're running a different Linux distribution or version.

Prerequisites

My WSL Debian container required two changes before I could install microk8s. I needed to enable systemd and install Canonical's Snap package manager.

Enable systemd

You can enable systemd by creating (or adding to) the file /etc/wsl.conf:

[boot]
systemd=true

You'll need to restart WSL after this:

Close your WSL terminal window
Run wsl.exe --shutdown from a PowerShell terminal
Re-open your WSL terminal window

Install Snap

Now that systemd is up, install Snap with the following set of commands:

sudo apt update
sudo apt install snapd

Get microk8s

To install microk8s:

sudo snap install microk8s --classic

System Setup

I use zsh, so I needed to:

add Snap's bin folder to my path
add an alias for microk8s.

I edited my ~/.zshrc file to source the alias file and add Snap's bin folder to my path:

source ~/.zsh_aliases
PATH=$PATH:/snap/bin

I created an alias file ~/.zsh_aliases and added the contents:

alias m8s'microk8s'
alias k8s='m8s kubectl'

Finally, I sourced my changes back into the current console session with the following:

source ~/.zshrc

To verify everything is working, issue the command:

k8s get nodes

You should see something like the following:

NAME            STATUS   ROLES    AGE   VERSION
computer-name   Ready    <none>   9h    v1.26.0

Up Next

Now that I have a working cluster, the next thing I'll write about is to add some essential features like role based authentication (RBAC) and a dashboard.

References

Kubernetes on Windows with microk8s and wsl2

How To Enable systemd in WSL

Installing Snap on Debian