Kubernetes, are you ready for production?

Last August, during my Kubernetes talk, I promised to share when the DZone Kubernetes trend report I wrote for will be published.

And ... it's finally out 🎉 and ready and free!

So, picture this: you can’t stop talking about Kubernetes. After some initial discussions, your company’s stakeholders have finally given you the green light to adopt it. You are driving a technological change in
your organization, which is very different than running a demo or POC(proof of concept). You are accountable for a huge product’s success. How do you proceed? What do you do? We’ve got you covered.

---

Table of contents

1. Download the free 2019 Kubernetes trend report
2. What is covered there
3. Footnote

---

Download the free 2019 Kubernetes trend report

You are welcome to download the free trend report from DZone, using this link.
You will be requested for a name and an email.
Not comfortable with sharing this information? Ping me on Twitter and we'll work it out.

---

What is covered in the article I wrote there (On Pg. 16) :

• Create your team of experts
• How to share cluster resource with multiple developers teams
• Securing virtual machines
• Access management
• Monitoring of Kubernetes cluster
• High Availability and Disaster Recovery
• Deployment

---

Footnote

• If you are totally new to Kubernetes, than before reading the article, it's highly recommended reading this and later on this.

• The article I wrote is on page 16 of the track report. It concludes how a Microsoft team boarded Kubernetes to production. This is only one piece out of great articles covering best practices of using Kubernetes in production.

---

💡 Are you working with Kubernetes in Production?
Share your best practices in the comments!

Comments

[Thorsten Hirsch]

I just started with k8s and used minikube for benchmarking redis (single instance, redis-benchmark with default settings). Here's a very, very short version of the result:

docker:                   100,000 requests/second
k8s/vm-driver=none:                 --- ERROR --- 
k8s/vm-driver=virtualbox:   7,500 requests/second

I wonder why VirtualBox is the default setting of minikube. I mean - the performance gap is huge compared to (native) docker performance! The performance with vm-driver=none would be even more interesting, because it would create docker containers as well, but under the control of minikube. Unfortunately I couldn't run this benchmark, because "sudo minikube start" resulted in a timeout of the dns service (I opened an issue).

[Thorsten Hirsch]

Now I could complete my benchmark:

docker:                   100,000 requests/second
k8s/vm-driver=none:        28,500 requests/second
k8s/vm-driver=virtualbox:   7,500 requests/second

Oh... that's less than I thought. Why is vm-driver=none so much slower than (native) docker? There are no additional instances of redis in k8s (I removed the 2 pods with the slave instances), so there's no redis replication happening that could harm the performance.

Is k8's load balancing (and all the other stuff) eating up 70% of the performance on my desktop computer? Is the performance decrease less on big server hardware?

[Adi Polak]

This is an interesting point. I know for a fact that when running things like Kafka and Spark you get better performance when using more than one server, balancing a big load of work. When used correctly the more data they process, the less time it takes them to process the next event/piece of data. When traditional solutions will probably break or fail.

Hence performing at best for big data/events streams at scale workloads.

BTW, Spark for a fact doesn't perform well on small loads, it takes more time than traditional operations on data.

I think that for K8s there is also an overhead when using it for one task/machine at a small load scale. Managing the pods, the networking between master and works/pods through the kubelet ( even when running locally).

I know Kafka and Spark are way different from K8s, but they rely on similar principals and challenges of distributed systems. Some of the challenges come from network bandwidth, lack of shared memory, distributed lock and much more. Which makes me wonder if ppl will be interested in reading a blog post about it.. 🤔

BTW, I like how your research shows that for one task, docker is good on its own. and for more. the vm= none, is none means bare metal? or actually native docker? from the documentation, it says that this is the vm supplied by the user.

BTW, documentation also say that When run in none mode, minikube has no built-in resource limit mechanism, which means you could deploy pods which would consume all of the host's resources. which can explain the better performance. What are the virtualbox limitations? can it be a low RAM? and than Radis need to write and read from disk and not from memory?

[Sean Allin Newell]

I'm currently at the forefront of k8s efforts at my company, and we're slowly starting a devops revolution that involves the whole company across all our departments (we have dedicated ops / build / release departments). Thanks so much for this!

[Adi Polak]

Hi, looking forward to hearing all about your progress!

[Sean Allin Newell]

Hi!

I just got ingress controllers to work in a sandbox cluster. Using traefik at the moment.

I may write a full blog post here on DEV about setting this all up and the challenges I'm running up against. The organization is excited, but there is a lot to consider.

[Adi Polak]

that's awesome! good luck and please share which process and architecture you have decided!
So many people will benefit from it :)

[Sean Allin Newell]

Definitely all still under discussion, but as I'm exploring here are a few constraints we have:

• minimal to 0 public cloud (le gasp! We use like, firebase for push notifs)
• on prem
• Enterprise support required (offerings like Pivotal Labs PKS)
• manual checks required (semi automated, traceability and auditability more important than speed and DX)
• rollbacks a must (ezpz)

So from this I'm thinking about:

• one 'config' mono repo with project specific files like
  • Dockerfiles
  • yaml files
  • configuration files
• PR based workflows
• tool like XebiaLabs Release and Deploy for workflow management
• use different K8s clusters for different environments (heard this from a talk)
• we use on prem splunk for logging, so we're looking into using Splunk Connect

Definitely still lots of unknowns for us and nothing is decided, but I'm at least pretty excited!

[Simon Massey]

Rollback is a single command at the end of the demo on the tutorial covered on this post

dev.to/simbo1905/git-driven-deploy...

on driving kubernetes from git webhooks.

[Adi Polak]

hi Simon, for some reason the link is broken :(

Hi Sean, regarding
*one 'config' mono repo

In order to help the dev team create new components without deep understanding of yaml files and docker files, we created dedicated scripts for them. Some teams have a generic enough configuration and this helped to abstract away the K8s and docker complexity. Also it helped with knowledge bottleneck. of course we would love to ramp up everyone to be an experts on K8s and docker. but there is not enough time, or sometimes not everyone want's to focus on it. and that's totally fine.

Regarding rolleback, you have this by K8s itself. But some versions wasn't stable in the past, so it is recommended to double-check it for large scale of deployment. Sometimes it is easier to save old versions just for the sake of redeploying them as a rolleback. I know it's not the best solution, but it can help when things go totally wrong.

I am excited to learn more about your solution and process! I am sure it will be an insightful journey!

[Simon Massey]

Thanks I updated the link - it's a dev.to post by myself so hopefully easy to find if the link breaks

[Simon Massey]

Sure if you have a deployment then it has the ability to roll back to a previous deployment version. What the demo does is use Helm and Helmfile to create the deployment, configmaps, secrets etc declaratively. You define only what different in the current deployment from the last (the image path, the env vars) and helm generates the k8s yaml and releases it. Helm then lets you from rollback any updates if you broken anything.

Everything we need to run our entire company is in a git repo and when you push to git it fires a webhook that runs helmfile to set up the new ”application infrastructure” on k8s. This is known as infrastructure-as-code. When a team member wants to set up a whole new app they just copy the config of a working one, change the image paths, edit the env vars and does a git push. The infrastructure continuous deployment pipeline running in k8s responds to the git webhook by pulling from git and running all helm templates and doing the installs or updates. That creates the deployment that wasn't there before for a new app.

You might also enjoy my earlier post showing how we use a slack chatbot to actually move the images between environments. New devs don't need to learn anything they just tell the slackbot to build the code into an image or move an image into live. I open sourced the chatbot like everything else.