DEV Community: shubhamrai1993

Kubernetes for data scientists - Hosting predictions

shubhamrai1993 — Sat, 08 Oct 2022 02:48:56 +0000

In the last issue we went through the workflow of a data scientist and where exactly kubernetes can prove to be a useful base on which to build a platform for it.

In this issue let’s go through a simple example to gain hands on experience for the same.

Prerequisites

Docker - https://docs.docker.com/get-docker/ - To launch the cluster nodes as containers
kubectl - https://kubernetes.io/docs/tasks/tools/#kubectl - A cli tool to interact with a running kubernetes cluster.

Setting up a playground

Before starting we need a playground to perform the demo. We will set up a kubernetes cluster on the local machine to do this. Although a cluster should contain multiple nodes for fault tolerance and high availability, we will mimic that behaviour using an awesome tool kind (kubernetes-in-docker).

At the end of this section we will have multiple containers running with each container acting as a separate cluster node.

Installation

Follow the directions provided here

Test the installation by running

$ kind --version
kind version 0.14.0

Launching the cluster

Now we start a local cluster using kind. We will create one control plane and two worker nodes. It is possible to have multiple of both.

Kubernetes can have multiple control plane and worker nodes. All the centralised cluster management components live on the control plane nodes while user workload run on worker nodes. Read more here

First, create a kind configuration in a file called kind-config.yaml. You can find it here. This will define the structure of our cluster -
```
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
```
Here, we have defined three nodes with one in the role of control plane and other two as worker nodes.
Launch a cluster using this config. This might take some time. Make sure docker daemon is up on your system before executing this -
```
$ kind create cluster --config kind-config.yaml
...
Thanks for using kind! 😊
```

Lets run kubectl to make sure our cluster is up -

$ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:63122
CoreDNS is running at https://127.0.0.1:63122/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
...

This tells us that the cluster is indeed up. We can also see the individual containers acting as nodes by executing docker ps.

Architecture

With our cluster up, let’s look at a broad architecture of what we are about to provision.

Broadly speaking, we will host multiple replicas of our application inside the cluster and try to access them from outside with requests being load balanced across the different instances.

To achieve this, there are a few kubernetes specific terminology that we need to be aware of -

Pod - Pods are the smallest deployable units of computing that you can create and manage in Kubernetes. In our case, one instance of the application will be running inside one independent pod. These are ephemeral resources and control plane can move them across nodes if needed.
Deployment - A deployment is useful where we want to have more than one replicas for an application. Kubernetes tries to always maintain the number of replicas to be equal to what is provided in a deployment. We will create three identical replicas for our application.
Service - A service is useful to load balance across a set of pods running on the cluster. Since pods are essentially ephemeral and can be replaced at any time, service provides a stable interface to access the pods running behind it. We will use a service to test our application.

These three resources will enable us to host a scalable endpoint for serving our application.

Preparing the image

With the cluster up, we can now deploy an application and test it out. We will create an application using the popular iris classifier dataset.

Clone the repo

The repo is available at https://github.com/shubham-rai-tf/iris-classifier-kubernetes. It already contains the code for building and serving predictions at /iris/classify_iris endpoint using fastapi.

Building the docker image

We need to package this code in a docker image to prepare it for kubernetes. A Dockerfile is provided in the repo for doing that - here.

This Dockerfile specifies the image we will need to create a container hosting the prediction endpoints in a uvicorn server on port 5000. More details about the syntax is available here.

Execute this command to build a local image -

$ docker build . -t iris-classifier:poc
...
$ docker image ls
REPOSITORY       TAG  IMAGE ID      CREATED         SIZE
iris-classifier  poc  549913d5b1f9  12 seconds ago  737MB

We can see that the image has been successfully created with name iris-classifier and tag poc. We will now load this image into the cluster to use it inside the cluster

Loading the image

This step is only needed because we don’t have an image registry to pull the newly built image from. In production, the image should be hosted in a private registry like Dockerhub or AWS ECR and then pulled into the cluster directly

Execute this command to load the locally built image to the cluster -

$ kind load docker-image iris-classifier:poc
Image: "iris-classifier:poc" with ID "sha256:549913d5b1f9456a4beedc73e04c3c0ad70da8691a8745a6b56a4f483c4f0862" not yet present on node "kind-worker2", loading...
Image: "iris-classifier:poc" with ID "sha256:549913d5b1f9456a4beedc73e04c3c0ad70da8691a8745a6b56a4f483c4f0862" not yet present on node "kind-control-plane", loading...
...

You can verify that the images have been loaded by listing images inside any of the three containers -

$ docker exec -it kind-worker crictl images
IMAGE                                TAG    IMAGE ID         SIZE
docker.io/library/iris-classifier    poc    549913d5b1f94    753MB

Deploying to kubernetes

Kubernetes is essentially a declarative system. That means, we describe the contours of what we want to do and control plane components constantly drive the system towards reaching that state.

To implement the architecture we discussed earlier, we will describe our intent in the form of a yaml file which acts as a record of intent. In kubernetes parlance these are called manifests.

All kubernetes manifests have the following fields -

apiVersion - Multiple resources are grouped together in same api versions. This provides a standardised way of deprecating or promoting a resource across kubernetes versions.
kind - Identifies the exact object type that is to be created
metadata - Contains fields that act as metadata for the created object. The apiVersion, kind and metadata.name fields together identify a unique resource inside a namespace
spec - This field contains the specification for the object to be created. Every kind defines its own structure for this field with its own implementation.

We will use the manifests present in the repo in files within the manifests directory here.

It defines two kubernetes resources, Deployment and Service in deployment.yaml and service.yaml respectively. Let’s go through both the sections.

Deployment

apiVersion: apps/v1
kind: Deployment
spec:
    # number of replicas
  replicas: 3
  template:
    spec:
      containers:
            # name of the image
      - image: iris-classifier:poc
        name: iris-classifier

The deployment manifest in deployment.yaml majorly defines the pod spec we want to deploy in terms of the image name and the number of replicas. Once we apply this, kubernetes will constantly take steps to maintain the number of replicas to what we specify here.

Service

apiVersion: v1
kind: Service
spec:
    # Type of the service
  type: ClusterIP
  ports:
    # Port where the service will be accessible
  - port: 8080
        # Port on the container where the traffic is to be forwarded
    targetPort: 5000
    protocol: TCP
  selector:
    app: iris-classifier

The service manifest in service.yaml defines how to load balance across the replicas created by the deployment. Here we have defined how the port on service is to be mapped to port on the containers. Since our application runs at port 5000, the targetPort is set to 5000. The service is exposed on port 8080. TCP traffic sent to 8080 will be load balanced across port 5000 on the containers.

Applying the manifests

Run the following command to apply the manifests to kubernetes -

$ kubectl apply -f manifests/
deployment.apps/iris-classifier created
service/iris-classifier created

Both resources are successfully created on the cluster. We can verify running the following commands -

$ kubectl get service iris-classifier
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
iris-classifier   ClusterIP   10.96.107.238   <none>        8080/TCP   37m
$ kubectl get deployment iris-classifier
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
iris-classifier   3/3     3            3           38m
$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
iris-classifier-5d97498ff9-77wqw   1/1     Running   0          39m
iris-classifier-5d97498ff9-8twjm   1/1     Running   0          39m
iris-classifier-5d97498ff9-znrz8   1/1     Running   0          39m

As we can see, the service is exposed at port 8080 and three pods have been created as we specified.

Modify deployment.yaml to have 2 replicas instead of 3 and re-apply. Kubernetes will delete one of the replicas to match the spec.

Making the prediction

Now that the resources have been created in the cluster, we can verify our deployment by calling the model using the service endpoint. Since we are using a local setup, we will have to port-forward the service to a port on the local machine.

In a cloud provider setup, this service will be bound to an external load balancer which can be accessed from the internet if needed.

Run the following command to perform port forwarding for the service -

$ kubectl port-forward services/iris-classifier 8080
Forwarding from 127.0.0.1:8080 -> 5000
Forwarding from [::1]:8080 -> 5000

We can verify by calling the /healthcheck endpoint on the model -

$ curl 'http://localhost:8080/healthcheck'
"Iris classifier is ready!"

To perform a test prediction, we will send a sample input to get a prediction -

$ curl 'http://localhost:8080/iris/classify_iris' -X POST \
    -H 'Content-Type: application/json' \
    -d '{"sepal_length": 2, "sepal_width": 4, "petal_length": 2, "petal_width": 4}'
{"class":"setosa","probability":0.99}

We get a prediction of class setosa with a 99% probability. By running multiple of these predictions, we can verify that the requests are indeed being routed to different pods in a round robin fashion.

Cleanup

Let’s remove all the kubernetes resources we had installed first -

$ kubectl delete -f manifests/

This will cleanup all the kubernetes resources we had created in the previous sections. Now we can take down the cluster as well -

$ kind delete cluster
Deleting cluster "kind" ...

Conclusion

In this issue we went through how to host a model as a callable service in kubernetes. Although this was a toy example where we built a docker image locally and ran it on a cluster running on the same machine, a typical production setup works on similar principles. A lot can be achieved with just these two resources.

In the subsequent issues we will explore other more advanced features like multi tenancy and access control which become essential as we move towards day 2 operations.

Container networking: A tour

shubhamrai1993 — Sun, 24 May 2020 14:21:35 +0000

Recently I was trying to move an application from one machine to another since the free trial for the earlier cloud provider was getting expired 😅. I took this opportunity to containerize this application and all it's upstream dependencies and learn something new along the way. I soon realised that containers, especially when it comes to networking, don't quite work the same way as that of a process. There are nuances to how you want to go about setting up your application deployments. There are multiple options to choose from which makes it all the more worse for a newbie. Let's go through those one at a time.

We are not considering the docker swarm specific options here. That was not my use case and since the arrival of Kubernetes you have a better alternative.

Docker provides several networking drivers for you to start your container with. But first let's find out where to look at all the networks that the docker has created for you.

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
4071b98202b2        bridge              bridge              local
badd725b14a0        host                host                local
97a49ff283fd        none                null                local

You can see that there are three types of networks docker daemon has already created for you. Each of these have their own names and network id. We will go into the various drivers and how to use them next.

Docker network drivers

Docker has six networking options you can start your container with

1. Bridge networks

Bridge is a link layer device that forwards traffic between two network segments. It provides an interface through which multiple disparate network segments can interact with each other and behave as a single cohesive unit. Docker uses the same idea to create a software bridge network that provides isolation between the docker host (the machine where the docker daemon is running) and the network that all the containers are running on. A bridge is created as soon as the docker daemon starts and all the containers launched on this host connect to this bridge by default. You can get the details for a network using it's name or id -

$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "15f169f892264417a374f774cd48a332831e64d89f9e1182e958b47183d675c2",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
    }
]

The thing to note here is IP Address Management (IPAM) config. All the containers attached to this network get an IP addresse from this subnet.

Docker also provides you an option to create your own bridge network if required. These are called User Defined Bridge networks. Let's look at how to create one quickly -

$ docker network create custom-network
34c9707787c96e777b3a2e0a2fac6284dd59da9753e3609c4007686b2e91fbbd

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
4071b98202b2        bridge              bridge              local
34c9707787c9        custom-network      bridge              local
badd725b14a0        host                host                local
97a49ff283fd        none                null                local

You can see that a new bridge network named custom network has been defined for you. By default it uses the bridge driver so we don't need to specify that here. It has several advantages over the default network. User defined networks allow you to address another container on the same network using it's name along with the IP assigned to it. You also get better isolation for your containers since all other containers listen on the default bridge network. You can also configure your custom network to cater to your requirements which might have unintended effects on the default network.

2. Host networks

Docker allows you to leverage the network of the host machine i.e. the machine where docker daemon is running to launch containers directly. This is similar to launching an application on a host bound to a particular port. The container is assigned no IP address of it's own. You can address the application running inside the container directly using localhost. It is very useful when you want to just run an application similar to a local deployment. One significant limitation to this approach is that you can only use it from Linux.

To use this option, just specify it while launching your container -

$ docker run --network host -d nginx:latest
27b01648936d03ccb214dd59f0175a6fada94fb80da32fa7b485ee91c37f2ffc

$ curl localhost:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
-----------------------------------------

3. None

As the name suggests, this is the docker network that does not exist. It is used to provide absolute network isolation to containers by disabling its networking. No incoming or outgoing connections are allowed to and from such containers respectively. To use this option simply supply this with --network option.

$ docker run --network none -d nginx:latest
27b01648936d03ccb214dd59f0175a6fada94fb80da32fa7b485ee91c37f2ffc

You can inspect the container and verify that no IP address has been assigned to it.

4. Macvlan

This option enables you to launch containers with MAC addresses assigned to them. Your container behaves as an independent device once launched on this network and can be connected to by legacy applications that require connection to a physical device. The option might be useful in some cases but docker recommends avoiding this method if possible.

5. Overlay

This is a multi node network that enables you to create a distributed network over a set of hosts. It allows containers launched on separate hosts to communicate with each other. This option is mostly relevant to Docker Swarm so we won't go into detail here. But we will look into how Kubernetes handles a similar use case at the end.

Docker also provides option to integrate with a specific networking driver provided by an external party through a plugin based interface.

Networking from a container's perspective

Now let's take a slight detour and look at how networking looks from a container's perspective. This will be helpful when we take a look at the Kubernetes networking later.

A container is basically a collection of Linux isolation technologies that enables it to behave as an independent entity of it's own. We are concerned with the networking namespace here.

Every container is launched with a networking namespace of it's own by default. It means a separate IP address, routing table, network interface and other networking resources. The containers are initialized with a virtual networking interface connected to the underlying network. The container does not care about the specific network type it is connected to at the moment and can be shifted between networks while running. While initializing, a container is assigned it's own IP by the docker daemon. Docker daemon satisfies the need for a DHCP in this case. It has it's own port space independent from the host machine. Docker provides you a way to create a mapping between the two port spaces -

$ docker run -d -p 8080:80 nginx:latest
c5897567ef62f35c47757e5a511d60aefab2b6cf13eabeb8f4543a5ec29c35ab

$ curl localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
-----------------------------------------

Here the port 80 inside the container has been mapped to 8080 outside. This option won't work if you choose the host networking driver since the host and the container already share their port spaces.

You can also ask a container to share the networking namespace of an already running container while running it. This basically integrates the two containers at the networking level while being isolated in other aspects. Let's look at how to achieve this.

$ docker run -d --name nginx-1 nginx:latest
27b01648936d03ccb214dd59f0175a6fada94fb80da32fa7b485ee91c37f2ffc

$ docker run -it --network container:nginx-1 ubuntu
/# apt update; apt install -y curl
/# curl localhost:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
-----------------------------------------

We launched an nginx container named nginx-1 followed by an ubuntu container that shares its network. You can see that once inside the ubuntu container we can hit the nginx-1 server on localhost directly without addressing it through IP or container name. Both the containers share IP addresses and port spaces. Another container trying to use the same port already engaged by nginx-1 container will not be able to come up because of port conflict.

You can assign a specific IP address to a docker container at the time of running it. The DNS settings are inherited directly from the daemon and can be overridden piecemeal if required. Docker docs has a really comprehensive page on how to achieve some of these.

How does Kubernetes do networking

Pods are the basic building block of a kubernetes cluster. It is a collection of containers that logically belong together. This abstraction can be compared to a set of processes running on a single VM. All the containers in a pod can access each other using localhost. As you might have guessed this is achieved similarly to the last example we saw in the last section but there is a little more subtlety involved.

All the pods on a single node are connected to a single bridge network that kubernetes names cbr0 or Custom Bridge. Whenever a new pod is created, a container called pause is started. The only job of this container is to hold on to an IP address on the bridge network. All other containers coming up in this pod are configured to share the networking namespace of this particular container. And thus all containers in the same pod can access each other using localhost.

Conclusion

Containers and the technologies built over it have revolutionized the way people build and deploy applications. But the basics of networking that these shiny new technologies leverage is as useful as ever. It was illuminating for me to look into the basics of these technologies and I hope anyone that comes across this finds it helpful as well.