DEV Community

Discussion on: Kubernetes, are you ready for production?

Collapse
 
sirseanofloxley profile image
Sean Allin Newell

I'm currently at the forefront of k8s efforts at my company, and we're slowly starting a devops revolution that involves the whole company across all our departments (we have dedicated ops / build / release departments). Thanks so much for this!

Collapse
 
adipolak profile image
Adi Polak

that's awesome! good luck and please share which process and architecture you have decided!
So many people will benefit from it :)

Collapse
 
sirseanofloxley profile image
Sean Allin Newell • Edited

Definitely all still under discussion, but as I'm exploring here are a few constraints we have:

  • minimal to 0 public cloud (le gasp! We use like, firebase for push notifs)
  • on prem
  • Enterprise support required (offerings like Pivotal Labs PKS)
  • manual checks required (semi automated, traceability and auditability more important than speed and DX)
  • rollbacks a must (ezpz)

So from this I'm thinking about:

  • one 'config' mono repo with project specific files like
    • Dockerfiles
    • yaml files
    • configuration files
  • PR based workflows
  • tool like XebiaLabs Release and Deploy for workflow management
  • use different K8s clusters for different environments (heard this from a talk)
  • we use on prem splunk for logging, so we're looking into using Splunk Connect

Definitely still lots of unknowns for us and nothing is decided, but I'm at least pretty excited!

Thread Thread
 
simbo1905 profile image
Simon Massey • Edited

Rollback is a single command at the end of the demo on the tutorial covered on this post

dev.to/simbo1905/git-driven-deploy...

on driving kubernetes from git webhooks.

Thread Thread
 
adipolak profile image
Adi Polak • Edited

hi Simon, for some reason the link is broken :(

Hi Sean, regarding
*one 'config' mono repo

In order to help the dev team create new components without deep understanding of yaml files and docker files, we created dedicated scripts for them. Some teams have a generic enough configuration and this helped to abstract away the K8s and docker complexity. Also it helped with knowledge bottleneck. of course we would love to ramp up everyone to be an experts on K8s and docker. but there is not enough time, or sometimes not everyone want's to focus on it. and that's totally fine.

Regarding rolleback, you have this by K8s itself. But some versions wasn't stable in the past, so it is recommended to double-check it for large scale of deployment. Sometimes it is easier to save old versions just for the sake of redeploying them as a rolleback. I know it's not the best solution, but it can help when things go totally wrong.

I am excited to learn more about your solution and process! I am sure it will be an insightful journey!

Thread Thread
 
simbo1905 profile image
Simon Massey

Thanks I updated the link - it's a dev.to post by myself so hopefully easy to find if the link breaks

Thread Thread
 
simbo1905 profile image
Simon Massey • Edited

Sure if you have a deployment then it has the ability to roll back to a previous deployment version. What the demo does is use Helm and Helmfile to create the deployment, configmaps, secrets etc declaratively. You define only what different in the current deployment from the last (the image path, the env vars) and helm generates the k8s yaml and releases it. Helm then lets you from rollback any updates if you broken anything.

Everything we need to run our entire company is in a git repo and when you push to git it fires a webhook that runs helmfile to set up the new ”application infrastructure” on k8s. This is known as infrastructure-as-code. When a team member wants to set up a whole new app they just copy the config of a working one, change the image paths, edit the env vars and does a git push. The infrastructure continuous deployment pipeline running in k8s responds to the git webhook by pulling from git and running all helm templates and doing the installs or updates. That creates the deployment that wasn't there before for a new app.

You might also enjoy my earlier post showing how we use a slack chatbot to actually move the images between environments. New devs don't need to learn anything they just tell the slackbot to build the code into an image or move an image into live. I open sourced the chatbot like everything else.

Collapse
 
adipolak profile image
Adi Polak

Hi, looking forward to hearing all about your progress!

Collapse
 
sirseanofloxley profile image
Sean Allin Newell

Hi!

I just got ingress controllers to work in a sandbox cluster. Using traefik at the moment.

I may write a full blog post here on DEV about setting this all up and the challenges I'm running up against. The organization is excited, but there is a lot to consider.