DEV Community: Rastko Vukašinović

Internal and external connectivity in Kubernetes space

Rastko Vukašinović — Sun, 11 Aug 2019 22:54:37 +0000

Services and networking — from ClusterIP to Ingress

As you are making your way through all the stages of your app’s development and you (inevitably) get to consider using Kubernetes, it is time to understand how your app components connect to each other and to outside world when deployed to Kubernetes.

Knowledge you will get from this article also covers “services & networking” part of CKAD exam, which currently takes 13% of the certification exam curriculum.

Services

Kubernetes services provide networking between different components within the cluster and with outside world (open internet, other applications, networks etc).

There are different kinds of services, and here we’ll cover some:

NodePort — service that exposes Pod through port on the node
ClusterIP — service that creates virtual IP within the cluster to enable different services within the cluster to talk to each other.
LoadBalancer — creates (provisions) load balancer to a set of servers in kubernetes cluster.

NodePort

NodePort service maps (exposes) port on the Pod to a port on the Node. There are actually 3 ports involved in the process:

NodePort service exposes deployment (set of pods) to the outside of the k8s node
targetPort — port on the Pod (where your app listens). This is optional parameter, if not present, Port is taken
Port — port on the service itself (usually same as pod port)
NodePort — port on the node that is used to access web server externally. By standard, NodePort can be in the range between 30000 and 32767. This is optional parameter, if not present, random available port in valid range will be assigned.

Creating the NodePort service

apiVersion: v1
kind: Service
metadata:
 name: k8s-nodeport-service
spec:
 type: NodePort
 ports:
 - targetPort: 6379
   port: 6379
   NodePort: 30666
selector:
 db: redis

Linking pods to a service

Selectors are the way to refer (link) service to certain set of pods.
As set of pods gets selected based on the selector (in almost all cases, pods from same deployment), service starts sending traffic to all of them in random manner effectively acting as load balancer.

If mentioned pods are distributed across the nodes, service will be created across the nodes to be able to link all the pods. In case of multi node service, service exposes same port on all nodes.

ClusterIP

In case of application consisting of multiple tiers deployed to different sets of pods, way to establish communication between different tiers inside the cluster is necessary.

For example, we have:

5 pods of API number 1
2 pods of API number 2
1 pod of redis
10 pods of frontend app

Each of above mentioned 18 pods have their own distinct IP addresses, but making communication that way would be:

Unstable since pods can die and be recreated with new IP any time.
Inefficient since we would have to load-balance within the integration part of each app.

ClusterIP service provides us with unified interfaces to access each group of pods — it provides a group of pods with internal name/IP.

apiVersion: v1
kind: Service
metadata:
 name: api-1
spec:
 type: ClusterIP
ports:
- targetPort: 9090
  port: 9090
selector:
 app: api-1
 type: backend

ClusterIP is default type of service, so if service type is not specified, k8s assumes ClusterIP.

When this service gets created, other applications within the cluster can access the service through service IP or service name.

LoadBalancer

In short, LoadBalancer type of service is provisioning external load balancer in cloud space — depending on provider support.

Deployed load balancer will act as NodePort, but will have more advanced load balancing features and will also act as if you got additional proxy in front of NodePort in order to get new IP and some standard web port mapping (30666 > 80). As you see, it’s features position it as the main way to expose service directly to outside world.

Main downside of this approach is that any service you expose needs it’s own load balancer, which can, after a while, have significant impact on complexity and price.

Let’s briefly review the possibilities:

apiVersion: v1
kind: Service
metadata:  
 name: lb1
spec:  
 externalTrafficPolicy: Local  
 type: LoadBalancer  
 ports:  
 - port: 80    
   targetPort: 80    
   protocol: TCP    
   name: http  
 - port: 443    
   targetPort: 443    
   protocol: TCP    
   name: https  
 selector:    
   app: lb1

Above creates external load balancer and provisions all the networking setups needed for it to load balance traffic to nodes.

Note from k8s docs: With the new functionality, the external traffic will not be equally load balanced across pods, but rather equally balanced at the node level (because GCE/AWS and other external LB implementations do not have the ability for specifying the weight per node, they balance equally across all target nodes, disregarding the number of pods on each node).

If you want to add AWS ELB as external load balancer, you need to add following annotations to load balancer service metadata:

annotations:    
 service.beta.kubernetes.io/aws-load-balancer-backend-protocol:"tcp"    
 service.beta.kubernetes.io/aws-load-balancer-proxy-protocol:"*"

Ingress

Real life cluster setup

When getting into space where we are managing more than one web server with multiple different sets of pods, above mentioned services turn out to be quite complex to manage in most of the real life cases.

Let’s review example we had before — 2 APIs, redis and frontend, and imagine that APIs have more consumers then just frontend service so they need to be exposed to open internet.

Requirements are as following:

frontend lives on www.example.com
API 1 is search api at www.example.com/api/search
API 2 is general (everything else) api that lives on www.example.com/api

Setup needed using above services:

ClusterIP service to make components easily accessible to each other within the cluster.
NodePort service to expose some of the services outside of node, or maybe
LoadBalacer service if in the cloud, or
proxy server like nginx, to connect and route everything properly (30xxx ports to port 80, different services to paths on the proxy etc)
Deciding on where to do SSL implementation and maintaining it across

So

ClusterIP is necessary, we know it has to be there — it is the only one handling internal networking, so it is as simple as it can be.
External traffic however is different story, we have to set up at least one service per component plus one or multiple supplementary services (load balancers and proxies) in order to achieve requirements.

Number of configs / definitions to be maintained skyrockets, entropy rises, infrastructure setup drowns in complexity…

Solution

Kubernetes cluster has ingress as a solution to above complexity. Ingress is, essentially, layer 7 load balancer.

Layer 7 load balancer is name for type of load balancer that covers layers 5,6 and 7 of networking, which are session, presentation and application

Ingress can provide load balancing, SSL termination and name-based virtual hosting.

It covers HTTP, HTTPS.

For anything other then HTTP and HTTPS service will have to be published differently through special ingress setup or via a NodePort or LoadBalancer, but that is now a single place, one time configuration.

Ingress setup

In order to setup ingress we need two components:

Ingress controller — component that manages ingress based on provided rules
Ingress resources — Ingress HTTP rules

Ingress controller

There are few options you can choose from, among them nginx, GCE (google cloud) and Istio. Only two are officially supported by k8s for now — nginx and GCE.

We are going to go with nginx as the ingress controller solution. For this we, of course, need new deployment.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: nginx-ingress-controller
spec:
 replicas: 1
 selector:
  matchLabels:
   name: nginx-ingress
 template:
  metadata:
   labels:
    name: nginx-ingress
  spec:
   containers:
   - name: nginx-ingress-controller
     image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller
     args:
     - /nginx-ingress-controller
     - configMap=$(POD_NAMESPACE)/ingress-config
     env:
     - name: POD_NAME
       valueFrom:
         fieldRef:
          fieldPath: metadata.name
     - name: POD_NAMESPACE
       valueFrom:
        fieldRef:
         fieldPath: metadata.namespace
     ports:
     - name: http
       containerPort: 80
     - name: https
       containerPort: 443

Deploy ConfigMap in order to control ingress parameters easier:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration

Now, with basic deployment in place and ConfigMap to make it easier for us to control parameters of the ingress, we need to setup the service to expose ingress to open internet (or some other smaller network).

For this we setup node port service with proxy/load balancer on top (bare-metal /on-prem example) or load balancer service (Cloud example).

In both mentioned cases, there is a need for Layer 4 and Layer 7 load balancer:

NodePort and possibly custom load balancer on top as L4 and Ingress as L7.
LoadBalancer as L4 and Ingress as L7.

Layer 4 load balancer — Directing traffic from network layer based on IP addresses or TCP ports, also referred to as transport layer load balancer.

NodePort for ingress yaml, to illustrate the above:

apiVersion: v1
kind: Service
metadata:
 name: nginx-ingress
spec:
 type: NodePort
 ports:
 -targetPort: 80
  port: 80
  protocol: TCP
  name: http
 -targetPort: 433
 port: 433
 protocol: TCP
 name: https
selector:
 name: nginx-ingress

This NodePort service gets deployed to each node containing ingress deployment, and then load balancer distributes traffic between nodes

What separates ingress controller from regular proxy or load balancer is additional underlying functionality that monitors cluster for ingress resources and adjusts nginx accordingly. In order for ingress controller to be able to do this, service account with right permissions is needed.

apiVersion: v1
kind: ServiceAccount
matadata:
 name: nginx-ingress-serviceaccount

Above service account needs specific permissions on cluster and namespace in order for ingress to operate correctly, for particularities of permission setup on RBAC enabled cluster look at this document in nginx ingress official docs.

When we have all the permissions set up, we are ready to start working on our application ingress setup.

Ingress resources

Ingress resources configuration lets you fine tune incoming traffic (or fine-route).

Let’s first take simple API example. Assuming that we have just one set of pods deployed and exposed through service named simple-api-service on port 8080, we can create simple-api-ingress.yaml.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: simple-api-ingress
spec:
 backend:
  serviceName: simple-api-service
  servicePort: 8080

When we kubectl create -f simple-api-ingress.yaml we setup an ingress that routes all incoming traffic to simple-api-service.

Rules

Rules are providing configuration to route incoming data based on certain conditions. For example, routing traffic to different services within the cluster based on subdomain or a path.

Let us now get to initial example:

frontend lives on www.example.com and everything not /api
api 1 is search api at www.example.com/api/search
api 2 is general (everything else) api that lives on www.example.com/api

Since everything is on the same domain, we can handle it all through one rule:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: proper-api-ingress
spec:
 rules:
 -http:
   paths:
   -path: /api/search
    backend:
     serviceName: search-api-service
     servicePort: 8081
   -path: /api
    backend:
     serviceName: api-service
     servicePort: 8080
   -path: /
    backend:
     serviceName: frontend-service
     servicePort: 8082

There is also a default backend that is used to serve default pages (like 404s) and it can be deployed separately. In this case we will not need it since frontend will cover 404s.

Bonus — More rules, subdomains and routing

And, what if we changed the example to:

frontend lives on app.example.com
api 1 is search api at api.example.com/search
api 2 is general (everything else) api that lives on api.example.com

It is also possible, with the introduction of a new structure in the rule definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: proper-api-ingress
spec:
 rules:
 -host: api.example.com
  http:
   paths:
   -path: /search
    backend:
     serviceName: search-api-service
     servicePort: 8081
   -path: /
    backend:
     serviceName: api-service
     servicePort: 8080
 -host: app.example.com
  http:
   paths:
   -path: /
    backend:
     serviceName: frontend-service
     servicePort: 8082

Note (out of scope): You can notice from the last illustration that there are multiple ingress pods, which implies that ingress can scale, and it can. Ingress can be scaled like any other deployment, you can also have it auto scale based on internal or external metrics (external, like number of requests handled is probably the best choice).

Note 2 (out of scope): Ingress can, in some cases, be deployed as DaemonSet, to assure scale and distribution across the nodes.

Wrap

This was a first pass through the structure and usage of k8s services and networking capabilities that we need in order to structure communication inside and outside of the cluster.

I, as always, tried to provide to the point and battle tested guide to reality… What is written above should give you enough knowledge to deploy ingress and setup basic set of rules to route traffic to your app and give you context for further fine tuning of your setup.

Important piece of advice: Make sure to keep all the setups as a code in files, in your repo — infrastructure as a code is essential part of making your application reliable.

Local Kubernetes setup with minikube on Mac OS X

Rastko Vukašinović — Fri, 09 Aug 2019 23:00:23 +0000

Kubernetes, container registry, Helm…

Minikube is ideal tool to setup Kubernetes (k8s from now on) locally to test and experiment with your deployments.

In this guide I will try to help you get it up and running on your local machine, drop some tips on where and how particular stuff should be done and also make it helm capable (I assume when you use k8s that at some point you will want to learn about and use Helm, etcd, istio etc).

This is your local k8s environment scaffolding guide.

Minikube installation

Minikube works with virtual machine, and for this it can use various options depending on your preference and operating system. My preference in this case is Oracle’s VirtualBox.

You can use brew to install everything:

$ brew cask install virtualbox minikube

In this case you could get some kind of inconclusive installation error related to virtualbox installation, especially on Mojave and probably everything afterwards.

Whatever it says, it is most probably a new security feature in MacOS X that is in your way.

Go to System Preferences > Security & Privacy and on General screen you will see one (or few) messages about some software needing approval to install. You should carefully review the list if there is more than one and allow installation of software you need — in this case software by Oracle.

After that is done you can re-run the command above and when it is done you should be ready for next steps.

Running and accessing the cluster

Starting it would be as easy as

$ minikube start

In order to optimally utilize your local machine’s resources I would suggest stopping it when you do not need it any more… With VirtualBox in center of it, it will go through you laptop’s battery pretty quickly.

Starting it again later will get you back where you left off:

$ minikube stop

Kubernetes dashboard is also available to you (while minikube is running):

$ minikube dashboard

I will assume you have kubectl installed locally and that you are already using it for some remote clusters so you got multiple contexts. In this case, you need to list contexts and switch to minikube one (in following commands assuming default name that is, ofc, “minikube”)

$ kubectl config get-contexts
$ kubectl config use-context minikube

Now you are in the context of your local k8s cluster that runs on minikube and you can do all the k8s things in it.

Ingress controller

To run your deployments that have ingress (and I assume most of them will), you will need ingress add-on:

minikube addons enable ingress

Make sure that you setup ingress based on your local hosts. It basically means that whatever you set as host in your ingress rules needs to be set up in your /etc/hosts file:

[minikube ip] your.host

Where “[minikube ip]” should be replaced with actual minikube ip. It also works with multiple, space separated local hosts after minikube ip.
Here is shortcut to do it in bash:

$ echo "$(minikube ip) local.host" | sudo tee -a /etc/hosts

Container registry — i.e. Docker registry

Reality of the real container registry usage in local environment is a rough one, so I will provide easy, quick and dirty option that makes it quite easy to deploy your local work to your local k8s, but deprives you of really important experience of using proper container registry.

Local container registry

Get your local docker context to point to minikube context:

$ eval $(minikube docker-env)

To revert: $ eval $(docker-machine env -u)

When in minikube context, to start local docker registry:

$ docker run -d -p 5000:5000 --restart=always --name registry registry:2

So, now you have local registry to push stuff to (as long as your docker is in context of minikube).

You can now do:

$ docker build . -t <your_tag>
$ docker tag <your_tag> localhost:5000/<your_tag>:<version>

At this point you can use localhost:5000/: as image in your deployment and that is it.

Using remote container repo

To use remote container repo locally you need to provide way to authenticate, which is through k8s secrets.

For local secrets management for ECR, GCR and Docker registry I recommend using minikube addon called registry-creds. I do not consider it safe enough to be used anywhere but in local env.

$ minikube addons configure registry-creds
$ minikube addons enable registry-creds

Note on ECR setup: Make sure that, if you are setting it for AWS ECR, and you do not have role arn you want to use (you usually wont have and it is optional), you set it as something random as “changeme” or smt… It requires value, if you just press enter (since it is optional) deployment of creds pod will fail and make your life miserable.

In case of AWS ECR, that will let you pull from your repo directly setting url as container image and adding pull secret named awsecr-cred:

imagePullSecrets:
      - name: awsecr-cred

I have to note here that running this locally worked quite chaotically for me and every session was new experience and new hack to make it work… Not a happy path.

Helm

Helm is package manager for k8s, and is often used for configuration management across deployments. With high popularity of the tool and raising adoption, I want to end this guide with the note about adding helm to your local k8s env.

It is quite easy at this point, just have minikube up and:

$ brew install kubernetes-helm
$ helm init

This should be deprecated information pretty soon, but in current case helm uses backend called Tiller and that is what gets installed/deployed during helm init execution.

You should check tiller deployment with: $ kubectl describe deploy tiller-deploy — namespace=kube-system

Valuable read: https://docs.helm.sh/using_helm/

Now you have full k8s local environment able to accept all of your test deployments before you decide to put them in the cloud (or “raw iron” server somewhere).

DEV Community: Rastko Vukašinović

Internal and external connectivity in Kubernetes space

Services and networking — from ClusterIP to Ingress

Services

NodePort

Creating the NodePort service

Linking pods to a service

ClusterIP

LoadBalancer

Ingress

Real life cluster setup

So

Number of configs / definitions to be maintained skyrockets, entropy rises, infrastructure setup drowns in complexity…

Solution

Ingress setup

Ingress controller

Ingress resources

Rules

Bonus — More rules, subdomains and routing

Wrap

Local Kubernetes setup with minikube on Mac OS X

Kubernetes, container registry, Helm…

Minikube installation

Running and accessing the cluster

Ingress controller

Container registry — i.e. Docker registry

Local container registry

When in minikube context, to start local docker registry:

You can now do:

Using remote container repo

Helm

Now you have full k8s local environment able to accept all of your test deployments before you decide to put them in the cloud (or “raw iron” server somewhere).

HAPPY SCALING