Building a Physical Test K8s Cluster

holden karau — Thu, 19 Nov 2020 20:16:46 +0000

Building the Test Cluster

To ensure that the results between tests are as comparable as possible, I'm using a consistent hardware setup whenever possible. Rather than use a cloud provider I (with the help of Nova) set up a rack with a few different nodes. Using my own hardware allows me to avoid the noisy neighbor problem
with any performance numbers and gives me more control over simulating network partitions. A downside is that the environment is not as easily re-creatable.

Building the Rack

If I'm honest, a large part of my wanting to do this project is that ever since I was a small kid, I've always dreamed of running "proper" networking gear (expired CCNA club represent). I got a rack and some shelves. (I also got an avocado tree to put on top and a cute kubecuddle sticker for good luck)

It turns out that putting together a rack is not nearly as much like LEGO as I had imagined. Some of the shelves I got ended up being very heavy (and some did not fit), but thankfully Nova came to the rescue when things got too heavy for me to move.

After running the rack for about a day, I got a complaint from my neighbor about how loud the fan was, so I swapped it out for some quieter fans.

The Hosts

The hosts themselves are a mixture of machines. I picked up three Rasberry Pi 4Bs. I'm also running a Jetson Nano and three Jetson AGX Xavier's to allow me to experiment with GPU acceleration. To support any x86 only code, I also have a small refurbed x86 present.

For storage I scrounged up some of the free flash drives I've gotten from conferences over the years to act as storage. This initial set up was not very fast, so I added some inexpensive on-sale external SSD drives.

Setting up Kubernetes

Since I want to be able to swap between the different Python scaling tools easily, I chose Kubernetes as the base cluster layer rather than installing directly on the nodes. Since it is easy to deploy, I used K3s as the cluster manager. The biggest pain here was figuring out why the storage provisioning I was trying to use wasn't working, but thankfully Duffy came to the rescue, and we figured it out.

What's next?

Up next, I'll start exploring how the different tools work in this environment. At the very start, I'll just run through each tool's tutorials and simulate some network and node failures to see how resilient they are. Once I've got a better handle on how each tool works, I'm planning on exploring how each of them approaches the problem of scaling pandas operations. Once that's done, we can start to get in a lot deeper and see where each tool shines. If you are interested in following along, check out my Youtube Channel on open source programming where I will try and stream the process that goes into each post. You can also subscribe to the mailing list for notifications on this on my books when I get something working well enough to make a new post :)

Disclaimer

This blog does not represent any of my employers, past or present, and does not represent any of the software projects or foundations I'm involved with. I am one of the developers of Apache Spark and have some books published on the topic that may influence my views, but my views do not represent the project.

In as much as possible, I've used a common cluster environment for testing these different tools, although some parts have been easier to test out on Minikube.

A First (Brief) Look at Ray on Kubernetes

holden karau — Thu, 29 Oct 2020 20:56:27 +0000

A First (Brief) Look at Ray on Kubernetes

After my motorcycle/Vespa crash last year I took some time away from work. While I was out and trying to practice getting my typing speed back up, I decided to play with Ray, which was pretty cool. Ray comes out of the same¹ research lab that created the initial work that became the basis of Apache Spark. Like Spark, the primary authors have now started a company (Anyscale) to grow Ray. Unlike Spark, Ray is a Python first library and does not depend on the Java Virtual Machine (JVM) -- and as someone who's spent way more time than she would like getting the JVM and Python to play together, Ray and it's cohort seem quite promising.

Installing Ray

Installing Ray was fairly simple, especially due to its lack of JVM dependencies. The one weird thing I encountered while I was installing Ray is the fact that its developers decided to "vendor" Apache Arrow. This was disappointing because I'm interested in using Arrow to get some of these tools to play together and vendored libraries could make it a bit harder. I filed an issue with the ray-project folks, and they quickly responded that they were working on it and then resolved it, so this is something I want to come back to.

Running Ray on K8s

Since I had not yet built my dedicated test cluster, I decided to give Ray on Kubernetes a shot. The documentation had some room for improvement and I got lost a few times along the way, but on my second try a few days later using the nightly builds I managed to get it running.

Fault Tolerance

Fault tolerance is especially important in distributed systems like Spark and Ray since as we add more and more computers the chance of one of them failing, or having the network between them fail increases. Different distributed systems take different approaches to fault tolerance, Map-Reduce achieves its fault tolerance by using distributed persistent storage and Spark uses recompute on failures.²

Fault Tolerance Limitations

One of the things that really excites me about Ray is its actor model for state. This is really important for some machine learning algorithms, and in Spark, our limitations around handling state (like model weights) have made streaming machine learning algorithms very challenging. One of the big reasons for the limitations around how state is handled is fault tolerance.

To simulate a failure I created an actor and then killed the pod that was running the actor. Ray did not seem to have any automatic recovery here, which could be the right answer. In the future, I want to experiment and see if there is a way to pair Ray with a durable distributed database (or another system) to allow the recovery of actors.

I want to be clear: This is about the same as in Spark. Spark only³ allows state to accrue on the driver, and recovery of state on the failure of the driver requires some additional custom code.

What's next?

The ray-project looks really interesting. Along with Dask and other new Python-first tools we're entering a new era of options for scaling our Python ML code. Seeing Apache Arrow inside of Ray is reassuring since one of my considerations is how we can make our tools work together, and I think Arrow has the potential to serve as a bridge between the different parts of our ecosystem. Up next I'm going to try and set up Dask on my new K8s cluster, and then also re-create this initial experiment on physical hardware instead of Minikube. If you've got thoughts or suggestions for what you'd like to see next, please do send me an e-mail and file an issue against the webpage on GitHub.

You can also follow along with my streams around distributed computing and open-source on my YouTube channel. The two videos for this post are Installing & Poking at Ray and Trying the Ray Project on Kubernetes. This post originally appeared on my new blog "Scaling Python ML" - http://scalingpythonml.com/2020/08/16/poke-at-ray.html :)

If your interested in learning more about Ray and don't want to wait for me, there is a great collection of tutorials in the project.

Well… same-ish. It's technically a bit more complicated because of the way the professors choose to run their labs, but if you look at the advisors you'll notice a lot of overlap. ↩
Technically it's a bit more complicated, and Spark can use a hybrid of these two models. In some internal places (like it's ALS implementation and other iterative algorithms), Spark uses distributed persistent storage for fault tolerance. ↩
Streaming Spark is a bit different ↩

DEV Community: holden karau

Building a Physical Test K8s Cluster

Building the Test Cluster

Building the Rack

The Hosts

Setting up Kubernetes

What's next?

Disclaimer

A First (Brief) Look at Ray on Kubernetes

A First (Brief) Look at Ray on Kubernetes

Installing Ray

Running Ray on K8s

Fault Tolerance

Fault Tolerance Limitations

What's next?