DEV Community: André

Syncing Kubernetes services on MicroK8s for local development with Python

André — Sat, 25 Apr 2020 19:18:32 +0000

Disclaimer: This was something I did because I was bored. There are probably better ways to do this (please share them if you know any!), and your mileage most definitely will vary.

The situation

Kubernetes is a common tool for deploying applications and services, and tools like Minikube or MicroK8s are great for developing applications locally with it.

However, there are still situations where developing outside the cluster has its advantages. One being less overhead; not needing to sync files and directories, or attaching to remote debuggers. I like having as few layers as possible between my code and my debugger, and sometimes, even Docker development itself is a pain which I try to avoid.

This, though, means that the application won't be able to access services running in the cluster by their name. This is an issue especially for me, since one of the projects I'm working on makes use of Kubernetes to dynamically provision resources, making it sometimes a pain to work with locally, and the resources all have service names I need to access.

So, recently, I started researching solutions. Minikube is a good start, but since it is recommended to run it in a VM, it has too much overhead for some of the workflows I'm developing. So, I tried MicroK8s, which is great, since it runs locally, with sufficient isolation (it is installed as a Snap package on Linux). However, neither Minikube or MicroK8s have any solution to DNS "outside the cluster" (that I know of).

Some suggested solutions

Usually the solution to this (at least on Linux) boils down to:

Run a local DNS server, or dnsmasq, configured to point the service host names to your machine;
Manually edit your machine's hosts file (/etc/hosts) to keep things in sync yourself. For example, Minikube's Ingress guide explicitly suggests you edit your hosts file.

Neither is great. For example, using dnsmasq or a local DNS server may interfere with VPNs, meaning you may not be able to access your cluster, or your VPN's services, or both; while manually editing hosts files is a pain, and requires administrator privileges.

An alternative is to use something like Telepresence (which sounds really promising, mind), but if you're developing outside containers, it may also mess with your DNS settings in such a way that VPNs will not work either.

What I want

This is what I want:

Be able to develop and debug my application without requiring building an image and running it in a cluster (within reason);
Still be able to access the services which are deployed in the cluster by their name;
Not mess up the VPNs I need to access enterprise resources.

Since DNS is out of the question, I'm left with hosts files, so I'll use those.

Hosts files

A hosts file, briefly, is a text file containing a "table" of IP Address to Host Name(s), like this:

127.0.0.1 localhost
192.168.1.1 my-router

These files exist on Windows, macOS, and Linux, and once updated, they will be used for host name resolution, instead of your DNS server (note that there are exceptions to this, which I'll not cover here).

Types of Kubernetes Services

There are several ways of exposing a Deployment's containers with a Service. Here are some:

NodePort, which maps a port on the host to the port on the container (much like port forwarding, or the -p option for docker run)
ClusterIP, which assigns a cluster-local IP Address for that Deployment
LoadBalancer, which allows routing to multiple replicas of a Deployment's pods to distribute load.

There's also Ingress resources, which you can use to expose Services outside the cluster as specific host names. You should preferably use these if you want to expose services to the outside.

Demonstrating the problem

Assume you have an Nginx web server you want to expose:

$ kubectl create deployment nginx --image nginx
deployment.apps/nginx created

This will create a deployment and a Pod for Nginx:

$ kubectl get pods
NAME                                         READY   STATUS      RESTARTS   AGE
nginx-f89759699-2thhf                        1/1     Running     2          7h3m

To expose this deployment's Pods (to the cluster's pods and to the outside), we need a Service. We can create one like this:

$ kubectl expose deploy nginx --port 80 --name nginx-svc
service/nginx-svc exposed

You can now check that it was created:

$ kubectl get services
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
nginx-svc    ClusterIP   10.152.183.235   <none>        80/TCP    39s

If you type that IP Address (note that it will be different in your case), you should see the Nginx welcome page:

$ curl 10.152.183.235
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

Now, the same nginx-svc is available to other Pods on the clusters via its name. You can test this by running another Pod on the cluster:

$ kubectl run -i -t --rm --image alpine name
If you don't see a command prompt, try pressing enter.
/ # apk add curl
<snip>
/ # curl nginx-svc
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

But if you do it outside the Cluster (i.e., on the host):

$ curl nginx-svc
curl: (6) Could not resolve host: nginx-svc

It fails, as there is no DNS entry for the service name outside the cluster.

The solution

If you add the IP address of the service to the hosts file, it will work:

$ echo '10.152.183.235 nginx-svc' | sudo tee /etc/hosts
10.152.183.235 nginx-svc
$ curl nginx-svc
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

Mind that this works because MicroK8s sets up a network interface and routes to allow traffic from your host machine to the cluster, and vice versa.

Anyway, this is great, but it means that I need to edit the hosts file manually. That's boring, so let's automate this:

Automating hosts file synchronization

We need two things for each service: it's name, and it's IP address inside the cluster. I could use kubectl and Shell Scripts to retrieve the information I need, but I decided to resort to Python instead. This also allows me to use the kubernetes library. This can be installed with pip. After that, we can do something like this:

import kubernetes.config

# The first thing we need to do is configure the library with our credentials. This will load them from ~/.kube/config.
kubernetes.config.load_kube_config()

core = kubernetes.client.CoreV1Api()

# Get the services from the server (I'll assume the default namespace)
services = core.list_namespaced_service(namespace='default').items

for service in services:
    print(f"Service {service.metadata.name} has IP {service.spec.cluster_ip}")

Running this, we get:

Service kubernetes has IP 10.152.183.1
Service nginx-svc has IP 10.152.183.235

It's a good start! Now, we need to add this information to the hosts file. I'll do this in a very naive way to keep things simple, so:

import kubernetes.config

# The first thing we need to do is configure the library with our credentials. This will load them from ~/.kube/config.
kubernetes.config.load_kube_config()

core = kubernetes.client.CoreV1Api()

# Get the services from the server (I'll assume the default namespace)
services = core.list_namespaced_service(namespace='default').items

# Open the hosts file in append mode, so that we can write to it.
# ATTENTION: DO NOT RUN THIS, IT MIGHT DAMAGE YOUR HOSTS FILE!
with open('/etc/hosts', 'a') as hosts_writer:
    # Write an empty line, in case the file doesn't end with a new line (in case you do run this)
    hosts_writer.write('\n')

    # The format of the hosts file is [IP] [Host Name], so we can write to it like this:
    for service in services:
        hosts_writer.write(f'{service.spec.cluster_ip} {service.metadata.name}')

Again, do not run this! You might damage your hosts file, which can cause you trouble.

But, assuming you do manage to run this (sudo kind of breaks things with Kubernetes, thankfully), your hosts file should be updated. However, if you did manage to run this, do not run this repeatedly, as this will keep appending new lines to the hosts file.

To make this a bit safer, you could perhaps make the script mark added lines with a comment, and then remove them before adding new ones.

Going further

Having the basic version of the script, we can go a bit further:

Add support for Ingress resources, such that they are also accessible from the host (if they have a defined host, of course);
Running this periodically in the Cluster, with a Cron job;
Extra validation checks for sanity;
Add filters such that we only expose some hosts.

You can check what I did at this GitHub Repository.

Thanks for reading!

I'm going to be a teaching assistant, teaching web technlogogies. What should I know?

André — Wed, 24 Jan 2018 22:37:34 +0000

Hello!

(Note: this may be quite long, but I appreciate your feedback)

As you might have read from the title, I've been recently invited by my polytechnic institute to help assist them in lecturing the classes for the "Internet Technologies II" course. To give some backstory about this course (and the degree it's a part of), it's the second course (of two) that's directly related with web technologies.

The first course focuses mainly on the 3 'pillars' of front-end web development: HTML, CSS, and JavaScript, and things like DOM manipulation, events and querying.
It then moves on to jQuery and Bootstrap, explaining the things that can be done with them.
Finally, a brief introduction to Canvas and SVG (+ XML with JavaScript) is done, whose objective is more to develop the thought process, than the actual need to use these technologies directly.
The students are evaluated with a project that consists in a purely client-side application, usually sort of a "toolkit", with many "apps" that use the concepts lectured through the course (for example, a calculator, or a clock made with Canvas, or a color picker, or something like a calendar).

Not much is lectured about layout, when I attended the course, the only aspects that were lectured were table layouts and absolute positioning. I think that this has changed, but I haven't yet been able to talk to the professor to know more about this.

The second course (which is the one I'm going to help with) focuses on the back-end. Due to our partnership with Microsoft, and the market in Portugal, we focus on ASP.NET MVC (up to a few years ago, it was pure WebForms, but it changed due to the altering landscape).
Although I prefer to work with Node.js, I do agree that ASP.NET MVC is a good choice for the web framework to teach. It's powerful, well-structured, and well-documented. The objective of the course is to teach the students how to combine the contents of the first course, and combine them with the concepts of back-end routing to controllers and views, introducing models and CRUD concepts with Entity Framework and SQL Server, and finally, authentication and authorization.
Since this course is lectured by two different teachers, the back-end part only represents 2/3 of the hours of the course. The remaining third focuses on technologies such as JSON and AJAX (which are still fetched using ActiveX because the teacher doesn't set up a static file server), and more advanced DOM concepts.
In this course, the students have two projects: One focuses on the ASP.NET part, the other focuses on building an app that uses the remainder of the technologies lectured on the last third.

We only have about 14 weeks, each having 2 classes (a 2-hour one and a 3-hour one). Some classes are used for presentations, where the students present to the class what they've been working on, and so that possible mistakes can be caught early on (for example, making an "online store" application, but there is no support to save pictures for the products). In total, we have about 11-12 weeks that are used for actual lectures. This, times 5 hours a week, gives us a total of about 50-60 course hours, spread over a semester.

Knowing this, I think that the concepts taught on the 2-hour classes could be moved to the first course (perhaps in detriment of Canvas and SVG?), but perhaps for this year I'll leave them as-is (I'll use proper APIs though, not ActiveX).

Here are my ideas for this course:

I've worked with some students that graduated from this school, and they often don't know much about layout. I should introduce them to the positioning methods, Floats, Flexbox, and ideally, CSS Grids. Media Queries are also a good idea, I often find that they don't know how Bootstrap works.
I think that introducing them not just to get data using AJAX, I also want to show them how they can submit data with it. Perhaps I should show them Fetch? It's still not standardized as far as I know, but Promises are pretty much everywhere now, and perhaps this is a good proxy to introduce them.
Note that I must be careful not to introduce things that are too bleeding-edge, the harsh reality is that most enterprise clients in Portugal still use old versions of IE.
I often find that people don't know how to use the browser's Developer Tools. Perhaps I should show them how they work? Work with them in debugging the DOM, CSS, and JavaScript, in an app that's deliberately buggy?
I also want to show them some "intermediate" concepts of JavaScript. Sadly, on the first course, not much is taught, and some features are dubious at best (read: with). Thinks like Arrow Functions, new APIs (which?), and perhaps, Classes (they come with a Java background to this course).
Depending on how things evolve, I want to transition to the back-end, so that the topics can be taught with more time and detail, because there isn't much time to explain all the concepts of ASP.NET in ~30 hours. I'd like to go further into things like model/form validation, the basics of RESTful APIs, and going further into authorization, like returning subsets of data based on the user's privileges.
I think that the two projects should be merged into one.
Should I, if I had the time, show them the client-side frameworks that are "hot" right now? React, for example, is gaining a lot of traction in Portugal's enterprises.

Having read this (thanks!), is there anything you would change? I know that there is much that could be changed, but such things must be done one thing at a time. I'd also like to know what your recommendations are when it comes to lecturing classes.
I am well aware that knowing about the subject is sometimes not enough for us to be able to teach about it in an effective way, so how would you go about it, if you were doing it?