DEV Community: Rich Burroughs

Cyclops: Platform Engineering for the Rest of Us

Rich Burroughs — Mon, 03 Feb 2025 17:52:36 +0000

Platform engineering is possibly the biggest concept to take hold in infrastructure over the last 5+ years, and there’s a big reason why. For decades, application engineers have dealt with systems that have constantly thrown roadblocks and delays in their way. Platform engineers address this problem by building systems that enable self-service and provide useful abstractions that help those other engineers build and run their applications. As we know, enabling self-service helps both productivity and developer happiness.

However, building and maintaining a platform can be expensive, and not every organization has the budget for a dedicated platform team. For organizations with platform teams, there’s an ongoing tension between either building custom tools, using open source tools, or buying commercial solutions. Each of those options requires some kind of investment, but for many teams, building platforms largely composed of open source tools is a strong choice. The team doesn’t spend a lot on software licenses and isn’t stuck maintaining all the code.

In this post, we’ll look at Cyclops, an open source tool for building developer platforms. Cyclops lets teams deploy and manage applications using Helm charts as templates.

Prerequisites

The main requirement for this tutorial is a Kubernetes cluster. In this example, we’ll use kind to provision a cluster, but feel free to use a different method if you prefer. If you’re not already using kind, you can install it using the instructions in the docs. You also need a local Docker-compatible daemon to use kind, like Docker, Podman, or Colima.

You will also need kubectl. You can find instructions for installing kubectl in the Kubernetes docs.

Provision a cluster and install Cyclops

First, provision a cluster.

kind create cluster --name cyclops-demo

Your kube context should already be set to point to the kind cluster. You can test that you can connect by viewing the namespaces in the cluster.

kubectl get namespaces

Next, install Cyclops into your cluster with kubectl.

kubectl apply -f https://raw.githubusercontent.com/cyclops-ui/cyclops/v0.15.4/install/cyclops-install.yaml

As you’ll see from the output, Cyclops is composed of Kubernetes native objects.

customresourcedefinition.apiextensions.k8s.io/modules.cyclops-ui.com created
customresourcedefinition.apiextensions.k8s.io/templateauthrules.cyclops-ui.com created
customresourcedefinition.apiextensions.k8s.io/templatestores.cyclops-ui.com created
namespace/cyclops created
serviceaccount/cyclops-ctrl created
clusterrole.rbac.authorization.k8s.io/cyclops-ctrl created
clusterrolebinding.rbac.authorization.k8s.io/cyclops-ctrl created
deployment.apps/cyclops-ui created
service/cyclops-ui created
networkpolicy.networking.k8s.io/cyclops-ui created
deployment.apps/cyclops-ctrl created
service/cyclops-ctrl created
networkpolicy.networking.k8s.io/cyclops-ctrl created

There should now be two pods running in a new namespace called Cyclops. We can view them with:

kubectl get pods -n cyclops

You should see output like this:

NAME                            READY   STATUS    RESTARTS   AGE
cyclops-ctrl-7984df7589-wv4dw   1/1     Running   0          36s
cyclops-ui-64c4cdd7f7-fxnb7     1/1     Running   0          36s

The first pod handles the Cyclops API, manages the CRDs, and communicates with the Kubernetes API server. The second pod runs the Cyclops web UI.

Next, we will install a set of example templates that we will use to see how Cyclops works.

kubectl apply -f https://raw.githubusercontent.com/cyclops-ui/cyclops/v0.15.4/install/demo-templates.yaml

Finally, let’s connect to the Cyclops web UI. First, forward a port to it using kubectl.

kubectl port-forward svc/cyclops-ui 3000:3000 -n cyclops

Next, connect to the UI with your browser at http://localhost:3000/.

Deploy Niginx with a generic app template

For the first portion of the demo, we’ll deploy Nginx using Cyclops.

Open a new terminal window/tab and create a namespace called nginx.

kubectl create namespace nginx

Now, go back to the Cyclops UI tab in your browser. Click Add module in the upper right corner of the Cyclops screen. Think of a module in Cyclops as an application and all of the other Kubernetes resources required to run it.

Next, we’ll select a template. Cyclops templates are Helm charts, and Cyclops can use charts from GitHub repositories, Helm chart repositories, or OCI repositories.

Select app-template from the Template pull-down list. This generic template creates a Kubernetes deployment and service for an application.

For the module name, enter nginx. Click on Advanced. Select the nginx namespace from the Target namespace pull-down menu.

Click on General. You can see that some defaults have been populated, like the image name version. Leave them set to the defaults. Those values are being pulled from a values.yaml file in the Helm chart.

You’ll see some other configurable options under Scaling and Networking. You can also leave those set to the defaults. Scroll down and hit the Deploy button in the bottom right of the window.

On the next screen, you’ll see more information about the module and its status as it’s deployed. There’s a link to the template it was deployed from and information about the Kubernetes resources that have been created.

Click on nginx Deployment to see that one pod is running. We can confirm that info in the terminal by running this command:

kubectl get pods -n nginx

That output should match what you saw in Cyclops.

Go back to the Cyclops browser tab. Under Actions, you can see several buttons for making changes to a running module. Edit will let you change the configuration. Reconcile will re-create the resources. Rollback will revert to the previous version of the module if you have made changes. You can view the code for the module using the Module manifest button, and *Rendered manifes*t lets you view the Kubernetes YAML for the running resources.

Click Edit, and then change the number of replicas to 2 under Scaling. Then, hit the Deploy button.

Once the status goes back to green, hit the Rollback button. You’ll see a list of the module's previous generations (versions). There should just be one. Click the Rollback button for that generation, and you’ll see a diff of the changes that were made.

Click the OK button to perform the rollback.

Some teams won’t want to manage changes through the Cyclops UI, of course. You can instead use Cyclops with GitOps tools like Argo CD. There’s a GitHub repo with examples here.

Finally, let’s clean up. Click Delete, and type in the module name to confirm.

Deploy Redis with Cyclops

We’ve seen how to deploy an app with a custom Helm chart, but Cyclops can also deploy applications using existing Helm charts. Let’s look at how that works by deploying Redis using the Bitnami chart, which is included with Cyclops.

At the terminal, create a namespace called redis.

kubectl create namespace redis

In the Cyclops browser tab, click Templates in the left nav bar. Type “redis” in the search box to see the info for the installed template. The source for it is the Bitnami GitHub repo, and it’s coming from the main branch.

Now, click on Modules in the left nav and then Add module again. Select redis from the module pulldown list, and type in “redis-cache” for the module name. Click on Advanced and select redis from the list of namespaces.

Scroll down. You will see many other options that can be customized for the Redis install. Feel free to expand and view any that interest you, but leave them set to the defaults. Then, click the Deploy button in the bottom right.

It may take a bit for Kubernetes to spin up all the resources, but eventually, they should all go green.

We can confirm the pods are running with this command:

kubectl get pods -n redis

You should see the primary/master node and three replicas like this:

NAME               READY   STATUS    RESTARTS   AGE
redis-master-0     1/1     Running   0          4m6s
redis-replicas-0   1/1     Running   0          2m21s
redis-replicas-1   1/1     Running   0          3m10s
redis-replicas-2   1/1     Running   0          2m46s

We can use any Helm chart with Cyclops like this to allow users to deploy the applications they need to run. In the case of a complex module like this one, you can set default values that are needed for your team or even create a custom Helm chart that exposes fewer options.

That’s it for the tutorial. To clean up, you can delete the kind cluster.

kind delete cluster -n cyclops-demo

Conclusion

We’ve learned how to deploy applications with Cyclops using pre-existing Helm charts and custom ones, and we’ve seen how Cyclops allows teams to easily expose the abstractions they need for developers to deploy and manage their apps.

Whether you’re at an organization that’s not staffed for a dedicated platform team or your platform team would like an easy way to provide self-service for developers, Cyclops could be a great fit for your workflow.

Learn more

The Cyclops docs are a great place to start.
Check out their open-source repository and support them with a star.
There’s a community Discord if you’d like help or to give your feedback.

KubeCon Chicago Wrapup

Rich Burroughs — Tue, 21 Nov 2023 22:29:53 +0000

KubeCon + CloudNativeCon North America 2023 was held in Chicago from November 6-9. This was the third North American KubeCon since the start of the COVID-19 pandemic.

I wrote my first KubeCon wrapup post for KubeCon San Diego in 2019. If you've read the past wrapups, you'll know that I developed a specific style for them. I live-tweeted about the talks I attended and other happenings and then pulled in my tweets and others for the posts.

Given the weird place that X/Twitter is at and the engagement problems many people have noticed, I decided not to live tweet this time. I took notes and wrote up my thoughts afterward.

This was also my last KubeCon with Loft Labs. I've had a lot of fun talking about the company's tools (especially vCluster) with folks in the Kubernetes community, but it's time for a new challenge for me. I also really need a break. If you know someone in the cloud native space looking for a Developer Relations person, feel free to have them reach out to me as long as they can be a little patient with the start date. The best way to reach me is LinkedIn.

Also, if you'd like to view the videos for the talks I recommend they should be posted within a couple of weeks on the CNCF's YouTube channel.

Pre-Conference Activities

I spoke at KubeCon Detroit last year and at KubeCon Amsterdam in April, but I didn't have any talks accepted this time. I did present at two events before the conference proper, though.

My first talk was at Cloud Native Rejekts, one of my favorite community conferences. If you're unfamiliar with Rejekts, the idea is to give a space for people to present their ideas that weren't accepted for KubeCon. The speakers and talks are always very high quality, and a lot of my favorite people in the community show up. My talk was called Open Source Dev Containers with DevPod, and I had a lot of fun presenting it. I talked through the struggles involved with providing easy-to-use and repeatable dev environments and did a demo of DevPod.

Adrian Mout from Chainguard speaking at Cloud Native Rejekts

I also did a lightning talk about vCluster at Multi-TenancyCon, one of the KubeCon co-located events. I didn't have much time to attend the rest of the event, but multi-tenancy is a topic that really interests me. It was fun to speak at co-located events, and at some point the CNCF started giving free KubeCon registrations to the co-located event speakers, so that was nice.

The one downside of attending these events is that they make the KubeCon week much longer. In my case, the trip went from five days for the conference proper to eight days. It was worth it for me, but it's something to consider.

I also attended the Lightning talks on Monday evening, which were a lot of fun. One of my favorites was the talk about the CNCF's Deaf and Hard of Hearing Working Group. I wasn't aware of their work around making events and meetings more accessible, and it's fantastic. I also loved Tim Hockin's talk about the problems with the Service primitive in Kubernetes and the Gateway API.

This was my first time making it to the Lightning Talks, and I will be back in the future.

Day One - Tuesday

This time around, the KubeCon schedule changed from Wednesday-Friday to Tuesday-Thursday. I liked that change as it meant traveling home on Friday instead of on the weekend. Thank you CNCF for that.

Keynotes

The opening morning's keynotes focused a lot on running AI/ML workloads. I didn't sense as strong a theme at this KubeCon compared to some in the past, but if there was a central theme, that was it. My friend Joseph Sandoval did a panel on AI/ML, and Taylor Dolezal did a panel with end users.

My favorite part of these keynotes was the panel about sustainability called Environmental Sustainability in the Cloud Is Not a Mythical Creature, hosted by Frederick Kautz. This is a super important topic, and I'm always happy to hear about the advances in this area. The talk mentioned Kepler, a tool I've wanted to look at.

There were also updates from the CNCF's graduated projects, which now include Cilium and Istio.

The CNCF also put together a very nice In Memorium video with tributes to folks in the cloud native community who died this year. I was happy to see my friends Kris Nova and Carolyn Van Slyck included. They were both fantastic people who contributed to the community, and their losses were huge blows. There is a GitHub repo where you can add your memories of those folks.

A Practical Guide to eBPF Licensing: Or How I Learned to Stop Worrying and Love the GPL - Jef Spaleta & Bill Mulligan, Isovalent

I don't focus much on open source licensing, but I wanted to learn how it impacts projects using eBPF. Jeff and Bill both work at Isovalent, and they did a thorough job explaining the situation. The portions of eBPF projects that run in the kernel are required to use the GPL, but CNCF projects must use the Apache 2.0 license. The recommendation was to use the GPL for the kernel bits and Apache 2.0 for the parts of the software that run in userspace. The CNCF has made an exception for projects that use eBPF, allowing them to use the GPL for the code that runs in the kernel.

This was a situation that I wasn't aware of at all. If you are working with eBPF projects, this talk is worth watching.

When Is a Secure Connection Not Encrypted? and Other Stories - Liz Rice, Isovalent

Next, I went to another eBPF talk. If you have read my last few KubeCon wrapups, you know already that it's an interest of mine. I'm also a big fan of Liz's. We had a great conversation on my podcast Kube Cuddle last year.

This talk covered how Cilium and other service meshes handle encryption and identity. It's a bit hard for me to sum up because it was pretty technical. Networking also isn't my strongest area. But if you are interested in networking and Cilium, check this one out. Liz is a great speaker.

Demystifying Service Mesh: Separating Hype from Practicality - Brian Redmond & Ally Ford, Microsoft

This was an engaging introduction to service mesh. It focused on the features most meshes provide, like observability, tracing, traffic management, blue/green deploys, canary testing, A/B testing, and even fault injection.

You may have heard the term Progressive Delivery (coined by James Governor) used to refer to some of these practices, as well as things like feature flags. Progressive delivery practices allow teams to deploy more safely, and deployment safety is a huge factor in developing high-performing teams. If you've ever been on call, you understand. Service meshes make things like using canaries to test and having rollbacks easy to implement.

If you are new to service mesh and want to see what it can offer your team, check out this talk.

Service Mesh Battle Scars: Technology, Timing, and Tradeoffs - Keith Mattix, Microsoft; John Howard, Google; Lin Sun, solo.io; Thomas Graf, Isovalent; Flynn, Buoyant

Could I go to yet another talk about service mesh? Yes, I could. This one was very different, though. It was a panel hosted by Keith Mattix, and the panelists represented Cilium (Thomas), Istio (Lin and John), and Linkerd (Flynn).

The focus was on areas where the approaches of the tools differ, and things got a bit spicy at times (which was intended). Keith was a very entertaining moderator, and the panelists were all experts in their fields. If you are interested in the differences between these projects or like panels that get a bit contentious, this presentation is for you. I did enjoy it, and it was great to have something entertaining at the end of the day.

I decided to take it easy in the evening after the talks. This was already day five of my trip, and I was feeling it. These big conferences like KubeCon can be very draining, so I focus on pacing myself. If you are new to this kind of conference, it's important not to feel like you have to do everything possible. It's okay to take a break and recharge or to spend time on the hallway track instead of seeing a talk in every slot.

Day Two - Wednesday

Keynotes

The day two keynotes began with a talk from Hermanth Malla and Laurent Bernaille of Datadog, who talked through an incident that caused an almost 24-hour outage for Datadog. That outage would be rough for any application, but I'm sure many customers were caught without a backup method to observe their systems. I have a lot of respect for folks who will talk openly about outages like this and share learnings with the community.

Other highlights for me were a panel on inclusion and Jeremy Rickard from Microsoft talking about Long Term Support (LTS) for Kubernetes. A Kubernetes Enhancement Proposal (KEP) is open to change the supported period from 9 months to a year, which is a great idea.

The Community Awards have always been a favorite part of KubeCon for me. Those folks put a lot of time and energy into improving the community, and it's great to see them get recognized for it. You can see a list of the award winners in this CNCF blog post. Congratulations to all of them.

Winner of the 2023 Top Documentarian award, Divya Mohan

Learning Kubernetes by Chaos – Breaking a Kubernetes Cluster to Understand the Components - Ricardo Katz, VMware & Anderson Duboc, Google Cloud

This was one of my favorite talks of the conference. The premise was to fix a broken kind cluster bit by bit, and the speakers explained the different components of the cluster as they fixed them (apiserver, controller manager, scheduler, etc.). I don't want to spoil any of the jokes, so I will leave it by saying this was a brilliant combination of humor and education. I highly recommend watching it, especially to beginners.

Dungeons and Deployments: Leveling up in Kubernetes - Noah Abrahams, Oracle; Natali Vlatko, Cisco; Kat Cosgrove, Dell; Seth McCombs, AcuityMD

The other talk I saw on Wednesday was another one filled with jokes. In this one, the speakers explained some parts of Kubernetes by playing a tabletop role-playing game. There were a lot of jokes and plenty of puns that left people in the audience groaning. This one was heavier on the humor than the education, but it was a lot of fun.

Documentary Film - eBPF: Unlocking the Kernel

I was very much looking forward to the premiere of the new documentary about the creation and growth of eBPF. I loved the Kubernetes Documentary from the same filmmakers, but I knew the Kubernetes story better going in than I knew this one.

I was initially introduced to eBPF through Brendan Gregg, who was at Netflix back then. Brendan was posting on Twitter about the Linux performance flame graphs he generated with eBPF, and I saw him speak at SRECon about that topic. But it was several years later before I understood that eBPF can do much more, including networking and other observability.

I could go on a lot more about the film, but I think I will write a separate review of it soon. So, for now, I will leave off by saying that I recommend it for folks interested in eBFP and that you can watch it for free on YouTube.

After the film, I headed over with some of the Isovalent folks to their post-event party. I saw a lot of friends and had a great time. I feel fortunate that I've been able to connect with so many people in this community and learn from them, whether it's about open source tools or other aspects of what we do, like community and inclusion. Thanks to Isovalent for throwing such a fun party.

Day Three - Thursday

Keynotes

I was dragging by Thursday, day seven of my trip, so I missed the keynotes. I heard from multiple people that Tim Hockin's keynote was great, though, so I watched it afterward on the conference platform.

Tim's talk was called Kubernetes in the Second Decade. It was a very interesting look from a Kubernetes expert at what directions the project should take in the next ten years and what the challenges are. Tim covered topics like running AI/ML workloads, multi-cluster, complexity, and reliability. He introduced the concept of a complexity budget, which I loved. He said that there's a finite amount of complexity we can add to Kubernetes and that we need to say no to some things now so we can do other cool things later.

I strongly recommend watching Tim's talk when the videos are released.

Despite my final-day fatigue, I made it to a couple of sessions on Thursday.

Sidecar Containers Are Built-in to Kubernetes: What, How, and Why Now? - Todd Neal, Amazon & Sergey Kanzhelev, Google

Making sidecar containers first-class citizens is a change I wasn't aware of before I saw the KubeCon schedule. This talk explained how the sidecar containers work, and this talk should be helpful for many people as loads of us use sidecars.

The new sidecars are basically init containers that continue to run and are set to restart always. They start before the "main" container and end after it so that they will capture data like logs and metrics for the primary container’s entire lifecycle.

The feature is alpha in Kubernetes 1.28 and will be beta in 1.29. It’s super useful work from the SIG Node team. Big thanks to them.

Releasing Kubernetes and Beyond: Flexible and Fast Delivery of Packages - Grace Nguyen, University of Waterloo; Adolfo Garcia Veytia, Chainguard; John Anderson, Ditto

The final talk I saw at KubeCon Chicago was in the last slot of the day, and it was a presentation by members of the SIG Release team.

If you've read my past wrapup posts, you know that I have a lot of love for SIG Release. They do a lot for the Kubernetes project, and it's all very much in the "chop wood, carry water" vein. Release engineering tends not to get much attention until something goes wrong, and it's very challenging work. So I really appreciate the folks on this team.

The talk covered a lot of things that are happening with SIG Release. The team is still moving from the original Google infrastructure to the new infra for the project. It's great that Google donated so much, but the project doesn't want to be too dependent on one company. The team has also been working with SIG Docs on a release checklist.

If you want to get involved with SIG Release, they have a program where people can shadow the current members to learn. You can find more info here. From what I've heard, it's a great program.

Conclusion

That was it for my KubeCon Chicago. Overall, I had a great time and am glad I could attend. I would have loved to see more of Chicago (the only pizza I had was Detroit-style), but hopefully I can make it back there.

I think I enjoyed the April event in Amsterdam a bit more. It feels to me like the US events are not bouncing back from the pandemic as well as the European ones. San Diego, the last KubeCon before the pandemic, had something like 15,000 attendees. I heard that Chicago was more like 8,000 to 9,000 registered, but the crowd didn’t feel that big.

There are additional reasons for this besides COVID-19, like companies cutting travel budgets. I know some people who had to travel to Detroit last year on their own dimes, and that may still have been the case.

But even a KubeCon that's a bit smaller is a fantastic time for me. I got to see so many friends from the community and learn some things, too.

I don't know if I'll be in Paris next spring. It will depend a lot on where I'm working. But I should be able to attend the next North American event in Salt Lake City.

Managing Access to Kubernetes Clusters for Engineering Teams

Rich Burroughs — Mon, 07 Feb 2022 17:33:50 +0000

by Daniel Olaogun

Kubernetes is a container orchestration tool for managing, deploying, and scaling containerized applications. It helps engineering teams deploy and manage applications across multiple servers with fewer complexities. Some of its best-known features include:

Self-healing, which automatically restarts your application container if it crashes
Horizontal scaling your application up or down as the traffic load increases or decreases
Automatic rollouts and rollbacks for gradually deploying an updated version of your application or quickly rolling back to the previous version if you detect an issue

In Kubernetes, your containerized application is abstracted by a pod that can be replicated in a node or across many nodes. Nodes run your containerized applications and can contain one or multiple pods depending on the node resources. A set of nodes is called a cluster.

As your Kubernetes cluster grows, you may need help managing it. This means you’ll need the ability to add users to your cluster and provide the required permissions, among other tasks. In this article, you’ll learn how to manage access to your Kubernetes cluster as well as how to manage the users who are given this access.

Why You Need Kubernetes Clusters

A Kubernetes cluster contains control plane and worker nodes that work together to handle and distribute traffic from within and outside the cluster. Worker nodes are a set of virtual or physical machines that run your containerized applications, while the control plane node controls the worker nodes. The control plane node manages and maintains the desired state of the cluster, scheduling pods to worker nodes based on available resources. It also provides the API endpoint that users interact with.

The following are some important use cases for Kubernetes clusters.

Provisioning of Multiple Environments

During the development and release of your application, you need environments for development and testing as well as a separate production environment for the application release. A Kubernetes cluster provides these environments using namespaces.

Previously, managing multiple environments for a single application meant spinning up separate virtual machines for each environment. This was tedious, and managing the differences between the environments could be difficult. Kubernetes clusters simplify the process.

Running Multiple Deployments

Kubernetes clusters allow you to run multiple deployments of your applications, for example development, testing, and production deployments of the same application, in one cluster. You can also deploy multiple applications with different functionalities. However, these deployments will share the resources provided in the cluster.

Easy Scaling of Deployments

As your application traffic increases, the application consumes more resources. Increasing its resources ensures that the increased traffic won’t cause downtime. With Kubernetes, you can scale your application deployments by replicating them on multiple nodes in your cluster. This distributes the incoming traffic load across all the nodes running your application in the cluster.

Managing Access in Your Kubernetes Cluster

As your organization grows and the number of deployments in your Kubernetes cluster skyrockets, you need more users to help manage the cluster. However, you should also ensure that you can effectively manage the access of those users.

Access to the Kubernetes API is managed through authentication, authorization, and admission control. When a user makes a request through the API using a client such as kubectl, Kubernetes checks the user’s authenticity. If the user can’t be verified, the request is rejected; if Kubernetes can verify the user, the request moves to authorization, which confirms that the user has the required permission to initiate this request. If they do not, the request is rejected, but if the user has the right permissions, then the request moves to the final verification of admission control. Admission control ensures that the request is good (such as verifying that the container image you want to deploy is secure).

Granting Users Access

Other users must be recognized by Kubernetes in order to connect with your cluster. However, Kubernetes does not provide the ability to manage users out of the box. Users are generally managed outside of Kubernetes using services like Microsoft Active Directory, Okta OpenID Connect, AWS Identity and Access Management, and Loft.

Managed Kubernetes services such as Azure Kubernetes Service (AKS), Amazon Elastic Kubernetes Service (Amazon EKS), and Google Kubernetes Engine (GKE) incorporate their identity management system with Kubernetes services to manage and authenticate users.

If you have a self-managed Kubernetes cluster, there are multiple services available to manage user access:

Single sign-on with Loft
OpenID Connect with Okta
Kubernetes authentication through Dex
Access management using OpenSSL; note that you should send the kube-config file in a secure and encrypted manner to your authenticated users

You can also use the above options to handle user access for managed Kubernetes services. The Kubernetes documentation provides other authentication strategies for authenticating users in your cluster.

Managing Access

Once users have been granted access to your Kubernetes cluster, there are several strategies to best manage that access.

Providing Only Needed Permissions

Once you have successfully authenticated the users required in your cluster, give them just enough permissions to perform their duties. Depending on your team structure, it is not always a good idea for all users to have the same high-level access. Otherwise, a user might perform an operation without understanding its consequences, or you might lose access to your admin rights because your user privileges were changed by a malicious user.

There are different authorization modes in Kubernetes used for access control, including role-based access control (RBAC), attribute-based access control (ABAC), node, and webhook. However, RBAC is commonly used to implement user roles and permissions. For more information on how to implement RBAC in your Kubernetes cluster, check out the official documentation.

Decommissioning Users as Needed

When a user is no longer a part of your cluster team, you should delete the user from the cluster. This prevents the user from continuing to access the cluster and performing unauthorized activities.

Most of the user management platforms noted above allow you to remove users with ease.

Enabling Auditing

Kubernetes auditing logs all actions performed in your cluster sequentially. Auditing your cluster gives you this information:

What happened?
When did it happen?
Who initiated it?
On what did it happen?
Where was it observed?
From where was it initiated?
Where was it going?

When something unexpected happens in your Kubernetes cluster, the logs generated by the audit will guide you in getting to the root cause.

Kubernetes does not enable auditing by default; you must enable it yourself. Doing so is up to you and your team, but for the security of your Kubernetes cluster, enabling auditing is recommended.

Using Loft for Access Control

Loft is a platform built on top of Kubernetes that adds multitenancy and self-service capabilities, enabling you to control and manage access in your Kubernetes cluster. As previously noted, you can integrate Loft into your cluster to handle authentication and access control. Loft also integrates with many single sign-on (SSO) providers that you can use with your cluster.

In Kubernetes, non-admin users don’t have the privileges to list, create, or delete namespaces in a shared cluster. However, Loft offers a feature called spaces, a virtual abstraction of a Kubernetes namespace. Once a space is created, a corresponding namespace is created, and if the space is deleted, the namespace is deleted as well.

Loft also offers an auditing section similar to Kubernetes auditing, which records all operations and actions performed by users and applications using the Loft API in your Kubernetes cluster.

Best Practices

Remember that when configuring access in your Kubernetes cluster, there are some best practices you should follow:

Follow the principle of least privilege
Enable auditing in your cluster
Routinely check roles and permissions assigned to users
Remove users that are no longer relevant in your cluster

Conclusion

As you’ve learned, you have multiple options for granting and managing user access in your Kubernetes cluster, whether your cluster is self-managed or managed by a cloud provider. It’s important that you provide the right level of access to your different users and revoke that access when necessary. This way, you can ensure that your cluster is safe from misuse as you scale up your Kubernetes workflow.

If you need a third-party solution for managing access control and self-service in your Kubernetes cluster, consider Loft. It integrates well with cloud-native tools and can be used with kubectl or GitOps. Loft is easy to implement and offers several cost optimization features. You can request a demo to see what Loft can do for you.

Photo by Matt Seymour on Unsplash

Kubernetes Cost Monitoring with Prometheus & Grafana

Rich Burroughs — Wed, 06 Oct 2021 18:37:07 +0000

by Cameron Pavey

Regardless of the infrastructure you are running, it is always important to keep an eye on your costs. There have been enough horror stories of cloud billing getting out of control that teams should have some measures in place to keep an eye on the usage of these resources to avoid surprises. Beyond this, there are other benefits to monitoring the usage and cost of your infrastructure. Having access to this extra data might be informative further down the line when making decisions about upgrading or scaling your infrastructure.

In this article, you will learn how to set up Grafana and Prometheus to monitor your Kubernetes cluster. Like Kubernetes, Prometheus is a graduate of the Cloud Native Computing Foundation. It is an open-source monitoring system that integrates with many other tools and systems to collect data. Grafana, another open-source project, acts as a dashboard and visualizer for various data sources, including Prometheus, for which it boasts first-class support. With these two tools, you should be able to glean some helpful insights about the usage of your cluster.

Setting Up Prometheus Operator

While it is possible to install and manage Prometheus and Grafana independently as standalone applications and connect them after the fact, there is quite a lot of configuration boilerplate involved, which can all be abstracted away by using Prometheus Operator. Specifically, for this guide, you can use kube-prometheus-stack, a Helm chart that handles setting up Prometheus Operator, as well as Grafana. This Helm chart will give you a functional monitoring stack with minimal configuration required, making it an excellent way to experiment with these tools. It is also suitable for setting them up for a long-term deployment if you do not have any existing components for your monitoring stack.

Before getting started, make sure you have the prerequisites installed. If you are using kube-prometheus-stack, all you will need is a Kubernetes cluster and Helm 3.

It’s recommended to use a non-default namespace for these resources to make things easier to manage further down the line. You can create a new namespace like so:

kubectl create namespace monitoring

The chart you are going to install exists within the prometheus-community repo, so you’ll need to add that before you can install the chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Next, to see what charts are available in this newly added repo, you can use the search command:

helm search repo prometheus-community

This will give some output like so:

The one that you will want to install is kube-prometheus-stack. You can install that in the namespace you just created with the following command:

helm install prometheus-stack --namespace monitoring prometheus-community/kube-prometheus-stack

This command will run for a bit while it sets up all the required resources, but once the command completes, you can verify everything is present by running:

kubectl get pod --namespace=monitoring

This should produce some output like so:

Now you have a monitoring stack running on your cluster. As you can see, there are multiple components required for everything to work, and it would take quite a bit of manual configuration to get to this same point without using the Helm chart. From this output, the two Pods which you will want to take a look at are:

prometheus-stack-grafana-* - the Grafana dashboard
prometheus-prometheus-kube-prometheus-prometheus-0 - the Prometheus instance

By default, there is no ingress configured for these Pods, but to get going quickly, you can port-forward traffic from the machine you are running kubectl on, allowing you to access the Pods. For Grafana, you will need to port-forward port 3000, and for Prometheus, it will be 9090. You can do this with the following commands (which are long-lived, so you will need to run them in separate terminal panes/windows):

kubectl port-forward <prometheus-pod> 9090 --address=0.0.0.0
kubectl port-forward <grafana-pod> 3000 --address=0.0.0.0

With these two commands running, you can access Prometheus and Grafana by navigating to the associated port on the kubectl client machine. This is fine for exploratory purposes, but it would be a good idea to use proper ingresses to expose these services for a long-term deployment.

You should see the Prometheus web interface if you navigate to port 9090 on your kubectl client machine. From the nav menu at the top, select status > TSDB status. From this page, you can see an overview of the data stored within Prometheus’ DB. This is a good way to quickly check that things are working as expected because if Prometheus is misconfigured, there will likely be no data here. If it is working, it will quickly be populated with quite a lot of data, as you can see here:

Next, navigate to port 3000, and your browser should present you with the Grafana login screen. When installed as described above, the default username and password should be “admin” and “prom-operator”.

kube-prometheus-stack includes several pre-configured dashboards in its Grafana instance, which is a nice bonus, as you can see quite a lot of detailed information about your cluster straight out of the box. Once you are logged in, from the left-hand menu, select Dashboards > Manage to see the preconfigured dashboards.

Now that your Grafana instance has access to your cluster’s metrics, you can use this data to gather insights about your Kubernetes usage, and subsequently, its cost. None of the preconfigured dashboards are specifically about cost, but you can create your own with some effort.

Monitoring Your Cost

This step will likely take a bit of trial and error to find the right combination of metrics to track and the right formula to get the insights you are after. To get some quick and easy results, you can start by aggregating basic resource usage metrics for CPU, RAM, and storage. These values can then be multiplied by whatever your hourly rate is for the resource in question. For example, if you are using Google Cloud Platform’s GKE service, pricing information can be found here.

The preconfigured dashboards that come with kube-prometheus-stack are worth looking at because they will give you some context for getting meaningful insights out of your data. For example, by looking at the included “Kubernetes / Compute Resources / Cluster” dashboard, you can see how CPU usage is calculated using a metric called container_cpu_usage_seconds_total.

Create a new dashboard by selecting the Create > Dashboard option in the left-hand menu. Next, you can add a new panel and enter the following PromQL in the provided field:

(sum(container_cpu_usage_seconds_total) / 60 / 60 ) * 0.0445

This should aggregate the CPU usage of your containers and convert it to hours, which can then be multiplied by the CPU/hr cost from GCP to get something like this:

Because this is a new cluster with a minimal workload, the cost is still low, but you can see that it steadily increases at a consistent rate. Using this approach, you can experiment with the data available to you and build a collection of charts to monitor your cluster costs.

This approach has some benefits and limitations. On the plus side, all the tools required are free and easy enough to set up. You retain complete control of your data and can extend it however you like to create your ideal cost-monitoring system. The obvious drawback is that a fair bit of manual effort is involved (and even more so if you opt to set up Prometheus and Grafana manually instead of with the Helm chart). Furthermore, the quality of the insights you get will be directly linked to how much time and effort you invest in configuring and optimizing your dashboards.

Alternative Options

There are some alternatives and additions which you should consider when setting all this up. Just as the Helm chart saves a lot of effort when setting up Prometheus, there are some pre-built solutions for monitoring resource usage costs with these tools. One excellent option is Kubecost, which offers Grafana dashboards built specifically for this purpose. The dashboards come preconfigured with an opinionated setup for monitoring cluster cost with GCP, so some tweaking may be required to get it working with your specific setup, but it is certainly worth taking a look at to see how they set things up.

When it comes to managing your resource usage cost, monitoring is only half the puzzle. To actually get some benefit out of it, you need to have an actionable plan based on your data. This is where services like Loft.sh come in. Loft’s Kubernetes platform has features to help manage your resource costs. Of particular note is sleep mode, which can scale down your non-prod virtual clusters when not in use to save resources and, therefore, money.

There are countless options available for setting up cost monitoring for your cluster. There is generally something for everyone, depending on how much time and effort you want to invest in your monitoring solution up-front. One of the nice things about software is that it is generally easy to change, so it very well could be worth your while to implement a low-effort solution with preconfigured tools like Prometheus Operator and Kubecost to determine if there is value there for you. From this evaluation, you will be in a better position to make long-term decisions about the direction you want to take for your cost monitoring and Kubernetes resources.

Kubernetes Network Policies: A Practitioner's Guide

Rich Burroughs — Thu, 09 Sep 2021 19:12:27 +0000

by Levent Ogut

Providing security for our infrastructure and applications is a never-ending continuous process. This article will talk about security in Kubernetes clusters, traffic incoming and outgoing to/from the cluster, and the traffic within the cluster. Some organizations behave as if their own workloads can be malicious and design their security policies accordingly. In addition, in today's world, we all use third-party plugins, libraries, and pieces of code from external resources. Although this has been increasing productivity, this also brings many security concerns. Isolating the traffic incoming and outgoing to our applications to only what’s absolutely necessary is one of the best approaches there is.

Why We Need Network Policies

It is of paramount importance to secure the traffic in our clusters. By default, all pods can talk to all pods with no restriction. NetworkPolicy resource allows us to restrict the ingress and egress traffic to/from pods. For example, it provides the means to restrict the ingress traffic of a database pod to only backend pods, and it can restrict ingress traffic of a backend pod's traffic to a frontend application only. This way, we can secure our resources so that only legitimate traffic is allowed to/from the applications. An example would be limiting traffic so that our frontend pods can only connect to the backend application, so that an attacker who compromises the front end can’t directly access the database or any other pods.

The functionality of controlling traffic is typically achieved in networks by using firewalls (software or hardware). Here in Kubernetes that functionality is implemented by the network plugins and controlled by network policies. Note that network policies are not a replacement for firewalls.

Requirements for Implementing Network Policies

Kubernetes provides networking functionality by using network plugins. Unless you have a network plugin that can implement network policies, you will not be able to use the functionality. Please note that even if the API server accepts the network policy configuration, this doesn't mean that it will be implemented unless a controller understands and implements the policy. Several network plugins support network policies and much more.

Network Plugins

There are two types of network plugins:

CNI
Kubenet

CNI type plugins follow the Container Network Interface spec and are used by the community to create advanced featured plugins. On the other hand, Kubenet utilizes bridge and host-local CNI plugins and has basic features.

Several network plugins were developed from various organizations, including but not limited to Calico, Cilium, and Kube-Router. A complete list can be found in Cluster Networking documentation. These network plugins provide Network Policy implementation and more, such as advanced monitoring, L7 filtering, integration to cloud networks, etc.

While some network plugins use Netfilter/iptables in their underlying infrastructure, others use eBPF technology on the underlying data path. Netfilter/iptables is very mature and builtin into the kernel. On the other hand, eBPF allows you to change the functionality on the fly without kernel upgrade. Not being dependent on the kernel version has led some big players to use eBPF based network plugins on very large scales.

It is imperative to select the correct network plugin for your Kubernetes cluster(s). If you are using cloud providers for your Kubernetes setup (such as AWS, Azure, GCP), they might already have deployed a network plugin that supports network policies. Please check the cloud provider documentation for further details.

Writing & Applying Network Policies

Isolation

In a Kubernetes cluster, by default, all pods are non-isolated, meaning all ingress and egress traffic is allowed. Once a network policy is applied and has a matching selector, the pod becomes isolated, meaning the pod will reject all traffic that is not permitted by the aggregate of the network policies applied. The order of the policies is not important; an aggregate of the policies is applied.

Network Policy Resource Fields

Fields to define when writing network policies:

podSelector

podSelector selects a group of pods using a LabelSelector. If empty, it would select all pods in the namespace, so beware when using it.

policyTypes

policyTypes lists the type of rules that network policy includes. Value can be ingress, egress, or both.

ingress

ingress defines the rules that will be applied to the ingress traffic of the selected pod(s). If it is empty, it matches all the ingress traffic. If it is absent, it doesn't affect ingress traffic.

egress

egress defines the rules that will be applied to the egress traffic of the selected pod(s). If it is empty, it matches all the egress traffic. If it is absent, it doesn't affect egress traffic.

Egress Rules

An array of rules that would be applied to the traffic going out of the pod. It is defined with the following fields.

Fields:

ports: an array of NetworkPolicyPort (port, endport, protocol)
to: an array of NetworkPolicyPeer (ipBlock, namespaceSelector, podSelector)

Ingress Rules

An array of rules that would be applied to the traffic coming into the pod. It is defined with the following fields.

Fields:

from: an array of NetworkPolicyPeer (ipBlock, namespaceSelector, podSelector)
ports: an array of NetworkPolicyPort (port, endport, protocol)

Walkthrough

Let's do a walkthrough of a network policy defined as below.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: network-policy-walkthrough-db
spec:
  podSelector:
    matchLabels:
      component: database
  policyTypes:
  - Ingress
  ingress:
  - from:
    - ipBlock:
        cidr: 192.168.1.2/32
    - namespaceSelector:
        matchLabels:
          team: dba
    - podSelector:
        matchLabels:
          component: backend
    ports:
    - protocol: TCP
      port: 5432

This rule applies to all pods that have labels with component key and database value (component=database). Network policy affects only ingress traffic as defined in policyTypes\.

The tree ingress rule entries are evaluated with OR. Let's look at how Kubernetes interpreted the configuration using describe\ subcommand:

$ kubectl describe networkpolicy network-policy-walkthrough-db

Name:         network-policy-walkthrough-db
Namespace:    default
Created on:   2021-08-30 18:06:48 +0200 CEST
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     component=database
  Allowing ingress traffic:
    To Port: 5432/TCP
    From:
      IPBlock:
        CIDR: 192.168.1.2/32
        Except: 
    From:
      NamespaceSelector: team=dba
    From:
      PodSelector: component=backend
  Not affecting egress traffic
  Policy Types: Ingress

Host with IP "192.168.1.2", all pods in a namespace that have team label set to dba and all pods in the same namespace that has label component set to backend are allowed to reach on port 5432.

Examples

Default Deny Ingress

An all deny ingress rule with an empty podSelector (selecting all pods in the namespace) is a good starting point for a fresh cluster. You can then explicitly allow required traffic. As the podSelector is empty, it will continue to match new pods when they arrive.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress-policy
spec:
  podSelector: {}
  policyTypes:
  - Ingress

$ kubectl describe networkpolicies default-deny-ingress-policy

Name:         default-deny-ingress-policy
Namespace:    default
Created on:   2021-08-28 16:47:33 +0200 CEST
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)
  Allowing ingress traffic:
    <none> (Selected pods are isolated for ingress connectivity)
  Not affecting egress traffic
  Policy Types: Ingress

As you can see, Kubernetes interpreted our configuration as intended. All pods in the namespace are now isolated, no ingress traffic is allowed to the pods, and egress traffic is not affected.

Allow Access to a Group of Pods from Another Namespace

In this example, we will look at a network policy that allows debugging pods to connect to the application pods.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-debug
spec:
  podSelector:
    matchLabels:
      component: app
  ingress: 
  - from:
    - podSelector:
        matchLabels:
          component: debug
      namespaceSelector:
        matchLabels:
          space: monitoring
  policyTypes:
  - Ingress

Please note that here we have a single from rule. If we had put the namespaceSelector into its own rule, the meaning would change drastically; this is where podSelector and namespaceSelector are used together.

Let's check how Kubernetes interpreted the policy:

$ kubectl describe networkpolicy allow-debug

Name:         allow-debug
Namespace:    default
Created on:   2021-08-30 22:36:48 +0200 CEST
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     component=app
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From:
      NamespaceSelector: space=monitoring
      PodSelector: component=debug
  Not affecting egress traffic
  Policy Types: Ingress

Here we only allow ingress traffic from pods with a label component set to debug in the namespaces with the label space set to monitoring.

Monitoring Network Policies

Monitoring the network policies and their behavior is an essential part of the deployment. Kubernetes offers the kubectl describe networkpolicy <NETWORK_POLICY_NAME> command to see how Kubernetes interpreted the network policy configuration. For detailed analysis, check out the network plugin's tools. Here we have a Kubernetes cluster with Cilium network plugin. Cilium offers a CLI tool, and from there, we can monitor the packets.

Let's get the IP address of our pod:

$ kubectl get pods -o wide

NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
nginx-deployment-66b6c48dd5-frsv9   1/1     Running   0          24m   10.0.0.136   valhalla   <none>           <none>

Let's get the endpoint id (in Cilium) of the pod:

$ cilium endpoint list

ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])  IPv6   IPv4         STATUS    ENFORCEMENT        ENFORCEMENT                                                                                                                   
5          Enabled            Disabled          50873      k8s:app=nginx                       10.0.0.136   ready   
...

We will monitor all traffic that goes to and comes from the endpoint with id 5.

Press Ctrl-C to quit

level=info msg="Initializing dissection cache..." subsys=monitor
Policy verdict log: flow 0xf9da54c5 local EP ID 5, remote ID 1, dst port 80, proto 6, ingress true, action allow, match L3-Only, 10.0.0.147:39772 -> 10.0.0.136:80 tcp SYN
-> endpoint 5 flow 0xf9da54c5 identity 1->50873 state new ifindex lxc4eced79e6ca0 orig-ip 10.0.0.147: 10.0.0.147:39772 -> 10.0.0.136:80 tcp SYN
-> stack flow 0xbbd5210b identity 50873->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.0.0.136:80 -> 10.0.0.147:39772 tcp SYN, ACK
-> endpoint 5 flow 0xf9da54c5 identity 1->50873 state established ifindex lxc4eced79e6ca0 orig-ip 10.0.0.147: 10.0.0.147:39772 -> 10.0.0.136:80 tcp ACK
-> endpoint 5 flow 0xf9da54c5 identity 1->50873 state established ifindex lxc4eced79e6ca0 orig-ip 10.0.0.147: 10.0.0.147:39772 -> 10.0.0.136:80 tcp ACK
-> stack flow 0xbbd5210b identity 50873->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.0.0.136:80 -> 10.0.0.147:39772 tcp ACK
-> endpoint 5 flow 0xf9da54c5 identity 1->50873 state established ifindex lxc4eced79e6ca0 orig-ip 10.0.0.147: 10.0.0.147:39772 -> 10.0.0.136:80 tcp ACK, FIN
-> stack flow 0xbbd5210b identity 50873->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.0.0.136:80 -> 10.0.0.147:39772 tcp ACK, FIN
-> endpoint 5 flow 0xf9da54c5 identity 1->50873 state established ifindex lxc4eced79e6ca0 orig-ip 10.0.0.147: 10.0.0.147:39772 -> 10.0.0.136:80 tcp ACK

We can see the traffic destined to our NGINX pod.

Policy verdict log: flow 0xf9da54c5 local EP ID 5, remote ID 1, dst port 80, proto 6, ingress true, action allow, match L3-Only, 10.0.0.147:39772 -> 10.0.0.136:80 tcp SYN

We can see the policy evaluation here.

Conclusion

We have explored why and how network policies are used within a Kubernetes cluster. Allowing only the required traffic is a security best practice, and Kubernetes allows us to implement this via declarative configuration of network policies. Since network policies depend heavily on the labels of pods/namespaces, it is straightforward to deploy rules that would also capture newly created resources.

It is highly recommended to test the network policies before applying them.

Observing traffic sources, destinations, and flows is imperative; as Kubernetes API does not include statistics, learning how to use the monitoring/troubleshooting tools of the network plugin becomes very important.

Folks at Cilium also developed a great UI Network Policy editor; make sure to check it out.

Docker Compose Alternatives for Kubernetes: DevSpace

Rich Burroughs — Thu, 09 Sep 2021 19:00:00 +0000

by Rich Burroughs

In this series, we’re looking at alternatives to using Docker Compose for building apps that run in Kubernetes clusters. While Compose is a handy way to stand up apps locally, there are advantages to running your apps in a Kubernetes environment while you develop. Your environment will be more like your production environment, and you can work with Kubernetes specific objects and manifests.

Previously in this series we’ve covered Tilt and Skaffold. Next up in the series is DevSpace, one of the other popular open source tools in the space. For transparency’s sake, I work for Loft Labs, which maintains DevSpace.

Architecture

DevSpace is client-only, and it has a lot of use cases for developing in Kubernetes. The client can be used to develop in local and remote Kubernetes clusters, build containers, and integrate nicely into your CI/CD pipelines.

DevSpace is configured with the devspace.yaml file and it supports port forwarding to connect to apps running in your cluster, as well as reverse port forwarding.

Installation

Since DevSpace is a single Go binary, installation is very simple. I’m on a Mac, so I installed it with Homebrew:

$ brew install devspace

There are other options in the installation instructions, like using npm, yarn, or just downloading the correct binary to your system.

Initializing Your Project

To set up your project to run with DevSpace, use the init subcommand:

$ devspace init

The DevSpace client will ask a few questions about deploying and building your project and set up a basic devspace.yaml file based on your answers.

There are several options for deploying your project with DevSpace, like using an existing Helm chart, Kubernetes manifests, existing Kustomize configuration files, or using the Component Helm Chart. This DevSpace feature builds a Helm chart on the fly based on your devspace.yaml file. If you want to try the DevSpace quickstart, using the Component Helm Chart is recommended.

For building your containers, you can use an existing Dockerfile or a custom build process. You can optionally ship the container images that are built to a registry. However, one of the most powerful features about DevSpace is that you can skip image building entirely and instead use the hot reloading which refreshes your running containers without having to rebuild the container image. We’ll explore this feature in more detail later on.

After running devspace init, you will have a devspace.yaml file to get started with, which you can further configure to suit your needs.

Entering Development Mode

To start developing with DevSpace, you first need to select the kube-context and namespace you will be working with. DevSpace makes this easy with the following commands:

$ devspace use context
$ devspace use namespace

Those two commands are convenient for people who work with multiple clusters and namespaces. Setting the namespace with devspace use namespace means you don’t have to pass it in with kubectl commands. But if you don’t want to use the interactive menu, you can pass in the context name and namespace as additional arguments to those commands (like devspace use namespace [NAMESPACE].

Once you’ve set your context and namespace, you enter development mode with this command:

$ devspace dev

This will get your project running in your cluster, as well as setting up any port forwarding that you’ve defined. At the end of the command, DevSpace will open a terminal similar to kubectl exec and you can start your application inside the container but still be able to access the application using your browser on localhost due to port-forwarding running in the background.

Hot Reloading For Containers

One of the significant features that DevSpace offers is container hot reloading. This means that DevSpace can update your running app without building a new container image each time you make a change. When you save a change in your code using your local IDE, DevSpace will automatically sync the changed files to your container that’s already running and can even be configured to (rebuild and) restart your app inside the container to pick up the changes.

Hot reloading can have a huge impact on developer productivity, as you can imagine. It provides faster feedback and improves cycle times, which also impacts developer satisfaction and happiness. You can try out hot reloading yourself using one of the quickstart projects, or you can see an example in this short YouTube video.

Developing Microservices with Kubernetes

When you’re working on a microservice, questions about how to handle service dependencies usually come up. If your service depends on APIs provided by an upstream service, do you need to run that other service too? And how do you manage it? This question often comes up in testing, too, regarding which dependencies to mock and which to run.

DevSpace makes it easy to set up multiple apps in different Git repositories that you’d like to run alongside your app. You can spin up the entire environment, including your dependencies, by running devspace dev. Docker Compose can’t work with multiple Git repos but with DevSpace you can set up a devspace.yaml in each Git repository and then add dependencies between the repositories, so DevSpace knows which services require each other when you start developing them.

Defining a dependency from one repo to another one is pretty straightforward in devspace.yaml as shown in this example:

dependencies:
- name: api-server 
  source:
    git: https://github.com/my-api-server
    branch: stable
  dev:
    ports: true

Hooks and Custom Commands

DevSpace offers a lot of extensibility, and two features that enable that are hooks and custom commands.

Hooks are actions that you want to occur in your build and deploy pipeline. Things you can do with hooks include executing commands, either on the local machine or in a container, uploading and downloading files, printing container logs, and more. Here’s an example of defining hooks in your devspace.yaml file, from the docs:

hooks:
# Execute the hook in a golang shell (cross operating system compatible)
- command: "echo before image building"
  when:
    before:
      images: all
# Execute the hook in a golang shell (cross operating system compatible)
- command: |
    echo Hello
    echo World
  when:
    before:
      images: image-1,image-2
# Execute the hook directly on the system (echo binary must exist)
- command: "echo"
  args: ["before image building"]
  when:
    before:
      images: image-1,image-2

The ability to chain actions together with hooks in your build/deploy processes is very powerful.

But what if you want to do other repeatable actions while developing your project? That’s where custom commands come in. You can create custom commands to do all kinds of tasks and then execute them at any time with the devspace run command, like:

$ devspace run [COMMAND NAME]

And remember how we talked about working with dependencies that are in other Git repos? You can run the custom commands for those dependencies that you’ve pulled in too. Let’s say you have a service with a database as dependency, one custom command could reset the database to a set of test data or it could execute a database migration, for example.

Hooks and custom commands allow you to tailor your DevSpace set up to your specific projects and languages and define your dev workflow as code. This is powerful, as it creates a shared workflow for your team that’s self-documenting.

Web UI For Kubernetes Development

While many teams prefer to use the DevSpace CLI to do a lot of their work, a web UI is also available to get a quick overview of what is running inside the current namespace.

With the DevSpace UI, you can view the logs of your running containers, view your build and deployment configurations, and view the custom commands you’ve defined in devspace.yaml. There’s also a link to open a terminal to a running container. You can do that from the command line, too, by running devspace enter.

Conclusion

DevSpace is a powerful tool that also is very flexible. People use it to automate even very complicated dev workflows, and hooks and custom commands allow you to fit it to your needs.

We talked earlier about the idea of encoding your dev workflows as code, and this is a compelling concept. Think about it as Infrastructure as Code but for your development workflow. Your devspace.yaml file becomes the source of truth for how you develop, and having your workflow defined in code makes onboarding new team members easier. It’s self-documenting.

All of the tools that we’ve looked at in this series are open source and free, and they all offer different benefits. If you’re developing apps that run in Kubernetes clusters, it would be worth your while to look at the tools in this space and find the one that works best for your team.

Loft Feature Spotlight: Sleep Mode

Rich Burroughs — Thu, 26 Aug 2021 16:25:57 +0000

It can be challenging to manage costs if your developers use Kubernetes clusters running in the cloud, whether they use shared clusters or have their own dedicated clusters. It’s difficult to keep track of what workloads are running where, and that gets even harder as you add clusters. Sure, you could rely on people to manually clean up after themselves, but we all know that nobody really likes to clean up. So you’re likely going to end up with a lot of idle containers and wasted resources. And besides increased costs and management headaches, wasting resources is terrible for the environment.

In many cases, you don’t need applications running in your clusters to be always available, especially when these applications run in dev clusters and engineers don't work 24 hours a day. Let’s do some quick math on this: Say that a typical engineer at your company works 40 hours a week. A week contains 168 hours. If the applications in their dev environment run all of the time, that leaves 168 - 40 hours = 128 hours a week where applications are up and ready to take traffic even if the engineer is not working. That equals roughly 76% ( 128 idle hours / 168 hours) of every week. And that’s assuming the engineer is actively using that app for 40 hours, which isn’t likely because they will also be in meetings, grabbing lunch or working on non-coding tasks.

What if I told you that you could automatically suspend workloads in your clusters when they’re not being used? Or even delete unused namespaces?

Sleep Mode is one of our favorite features in Loft because of the significant benefits that our customers get from it and because it's so easy to measure the financial impact. Sleep Mode can automatically scale down your apps when they’re not used to save on cloud resources and overall cost.

How Sleep Mode Works

Sleep mode works based on ReplicaSets. Let’s say for example that you have an NGINX Deployment that is set to run 5 replicas. When Loft detects that the namespace the app is in has been idle for a predefined amount of time, it will automatically scale the NGINX ReplicaSet down to 0 replicas, deleting all of the pods that belong to this NGINX Deployment. Loft remembers that there should be 5 replicas running. Once it detects activity to the namespace (e.g. a kubectl request such as kubect get pods), it will restore the previous number of 5 replicas, and Kubernetes will spin them back up again.

Loft detects whether the namespace is idle by examining incoming API requests. The Loft API Gateway acts as a proxy for the Kubernetes API server, so it can see when a request is made for a particular namespace. As soon as it sees an API request coming in, like a kubectl command, it will fire the pods back up. That’s the case for any other API requests, like from Helm or other tooling you have in place that uses the kube-context for any of the clusters you connect to Loft.

A developer could simply run kubectl get pods -n namespace as soon as they sit down at their laptop, and within a few seconds, they’d have their dev environment back up. Since nothing has been changed in their namespace besides the numbers of replicas running, they’ll quickly be back to the place they left off at. And without wasting resources while they were away.

Sleep mode works with both the classic Kubernetes namespaces and virtual clusters.

Enabling Sleep Mode

Let’s look at an example of enabling sleep mode for a user’s namespace. If you’d like to follow along and you’re not currently using Loft, you can run through the first two steps of our quickstart to get Loft installed and running in your cluster.

Add a user by clicking on the Users icon in the left navigation bar. Fill in the relevant information and hit the Create button.

Next, click on Clusters in the left navigation bar and then loft-cluster. Then click on the Accounts tab at the top of the screen.

Next, click on the user that you added. This is their Loft account in the Kubernetes cluster. The settings for sleep mode are under the Space Creation Settings. A space is a virtual representation of a self-service namespace that has additional functionality (like sleep mode).

Here you can adjust the number of minutes before sleep mode kicks in for this user’s spaces. You can also set a time to delete inactive spaces. Auto-delete is a great way to make sure that unused resources inside your clusters get cleaned up automatically.

You could use sleep mode and auto-delete in conjunction, like setting the user’s spaces to sleep after 60 minutes of inactivity and to be deleted after one month of inactivity. Or whatever combination of values works best for your users’ workflows. You can also define separate sleep mode and auto-delete timeout values for each individual namespace if needed.

And if you’re using virtual clusters, the process is the same. Each virtual cluster in Loft has a corresponding Loft space with the same name. You simply apply the sleep mode or auto-delete settings you want for the virtual cluster to either that space or the account that owns it.

You probably don’t want to edit every individual account to enable sleep mode. There are more options, like creating an account template that will enable sleep mode for all accounts created, or adding an annotation to accounts if you create them with YAML and kubectl. This is particularly useful if you use single sign-on for Loft and you want to auto-configure sleep mode and auto-delete for certain Active Directory or Okta user groups, for example.

There’s a lot of flexibility with how you can handle sleep mode, and there are more details in the docs.

Manually Triggering Sleep Mode

Users in your Loft-managed clusters can also trigger sleep mode manually. Just click on Spaces in the left menu, and click the sleep icon above the space.

If the user prefers the command line, they can also run:

loft sleep [SPACE_NAME]

Manually waking up spaces is just as easy. You can wake them up from the Spaces section of the web UI:

As we mentioned earlier, spaces will wake up automatically when they receive API requests, so you could also just run any kubectl command that touches that space, like:

kubectl get pods -n [SPACE_NAME]

Or use the Loft CLI to explicitly wake up a namespace:

loft wakeup [SPACE_NAME]

Conclusion

As you can see, there’s a lot of possibilities with sleep mode and auto-delete. Sleep mode can suspend workloads that aren’t being used, and auto-delete can automatically clean up idle Kubernetes namespaces and virtual clusters. These features can help you eliminate waste in your infrastructure and reduce some of the headaches that come with managing a large number of tenants operating in shared Kubernetes clusters.

At Loft Labs, we believe that self-service Kubernetes is essential, both for developers who don’t want to wait for infrastructure to be provisioned as well as for platform engineers who have huge backlogs and would rather not be responsible for creating every new namespace. With Loft, developers can provision namespaces and virtual clusters when needed, and platform engineers can ensure guardrails are in place to reduce waste. In the end, we all just want to do our jobs the best way we can, and self-service infrastructure is a critical part of making that happen.

PHP Laravel Development with Kubernetes using DevSpace - Developer Edition

Rich Burroughs — Fri, 20 Aug 2021 17:43:07 +0000

by Levent Ogut

Kubernetes is an excellent open-source container orchestration platform that brings automatic scaling, automatic recovery, observability, and many more features. Since it differs from traditional operations, it has changed the development and deployment workflows as well. Debugging an application on Kubernetes can be a challenge. DevSpace is a tool that helps you develop, deploy, troubleshoot simple or complex applications. We will use a Laravel project to demonstrate its features; Laravel is a popular framework used by the PHP community with great features like extensibility, inheritance, and reusability with high customization options.

Architecture

We will look at ways to deploy a Laravel based application into a Kubernetes cluster for development and production environments. We will develop our application while the application is running in Kubernetes as if we are developing locally. And we will be able to troubleshoot our application in real-time with ease.

The desired setup uses four containers, in three pods:

PHP-FPM container, which processes all the PHP assets.
Nginx container, which serves static files and acts as a reverse proxy for the PHP assets.
MySQL container, as the database.
Redis container, as session and cache.

Introduction to DevSpace

Continuous Delivery is a challenge while developing on Kubernetes. Without using a special tool, you need to build and deploy every time code or assets change. DevSpace handles this seamlessly either by synchronizing files and hot reload of the container in question or automatically rebuilding and deploying the image(s) required. DevSpace allows you to develop in a Kubernetes cluster as if you are developing in your local machine.

Feature Highlights

Agile development on local and remote Kubernetes clusters. Execution of entire continuous development and deployment pipeline, and a single command to deploy all components of your application.
Declarative configuration kept in source code, in the devspace.yaml file. All of the development, deployment, and pre/post-deployment actions are defined in a single file.
Hot Reloading for faster feedback. Instead of building and re-deploying artifacts, DevSpace allows you to use high-performance and bi-directional file synchronization. This allows changes to trigger a hot-reload on the deployed container. All of these features are highly configurable.
Extensibility. You can extend the functionality of DevSpace via the plugin system. Hooks and commands are also built-in constructs; you can expand the functionality heavily used in CI/CD pipelines.
Easy clean Up. You can delete the resources created via devspace purge\ in a single simple step.
Client Only. DevSpace doesn't require server/cluster side components. A single executable on a local machine is sufficient to develop, troubleshoot and deploy.

Requirements and Setting Up Development Environment

The following tools should be installed on your local development machine:

Kubectl, documentation here.
Helm, documentation here.
DevSpace, installation instructions here.

Developing with DevSpace

First, let's start with the code. Clone the repository to your local development machine as follows. This code includes a vanilla Laravel installation, a Dockerfile, and a devspace.yaml prepopulated.

$ git clone git@github.com:loft-sh/devspace-php-laravel-nginx.git

Copy .env.example to .env.

$ cp .env.example .env

Now open the .env file and modify where necessary, like adjusting port numbers if needed.

After this, we can generate the Laravel APP_KEY variable via the following command:

$ devspace run generate-key

Review the variables by running the devspace list vars command, and set variables where necessary.

$ devspace list vars

DevSpace will ask you a few questions regarding the image repository and other variables not defined in the .env file, and then show the output of the variables.

 Variable              Value                                                
 APP_DEBUG             true                                                 
 APP_IMAGE             leventogut/php-laravel-nginx-devspace                
 APP_KEY               xxxxxxxxxxxxxxxx  
 ASSET_VOLUME_NAME     static-asset-volume                                  
 ASSET_VOLUME_SIZE     1Gi                                                  
 DB_DATABASE           laravel                                              
 DB_HOST               mysql                                                
 DB_MYSQL_VERSION      8.0.23                                               
 DB_PASSWORD           xxxxxxxxxxxxxxxx                                     
 DB_PORT               3306                                                 
 DB_ROOT_PASSWORD      xxxxxxxxxxxxxxxx                                     
 DB_USERNAME           laravel                                              
 NGINX_CONFIG_HASH     740941                                               
 NGINX_IMAGE_VERSION   1.9                                                  
 REDIS_PASSWORD        xxxxxxxxxxxxxxxx                                    
 REDIS_VERSION         6.0.12

Running devspace dev

DevSpace is context-aware; it follows your Kubernetes config to determine the Kubernetes cluster to deploy on. However, It is good practice to set the context and namespace to use with:

$ devspace use context docker-desktop

[info]   Your kube-context has been updated to 'docker-desktop'
         To revert this operation, run: devspace use context maple-staging

[done] √ Successfully set kube-context to 'docker-desktop'

$ devspace use namespace laravel

[info]   The default namespace of your current kube-context 'docker-desktop' has been updated to 'laravel'
         To revert this operation, run: devspace use namespace 

[done] √ Successfully set default namespace to 'laravel'

Now that we have all the variables and the configs, we can start in-cluster development:

$ devspace dev

DevSpace will build the artifacts we have defined in the devspace.yaml, deploy all components, and start log streaming from the configured containers. This might take a few minutes.

In a few minutes, DevSpace will open a browser window showing a login screen. Previously, we have installed the laravel/ui package to test MySQL and Redis. Simply register as a new user, and you will be redirected to the index page. The index page has several links, including a link to the ping/pong route we will use in a few steps.

Workflow

At this stage, we have deployed our application into the Kubernetes cluster, and DevSpace is watching any changes on the project directory.

Now, having started DevSpace in development mode, we can change our code and see the immediate effect on our application that is running in the Kubernetes cluster.

Open the web.php file under the routes directory with your favorite editor. And paste the following code.

Route::get('ping', function () {
    return "pong";
})

Any asset added to the repo folder will also be synced (according to the sync rules defined) to the container.

At this stage, you can try adding controllers, routes, dependencies and observe the ease of development.

Port Forwarding and Reverse Port Forwarding

You can reach the application via port forwarding. These can be defined in the devspace.yaml file. In the current configuration, the Nginx container's port 80 is forwarded to local port 8080. The browser will be automatically opened after a successful deployment and start of the containers.

You can configure reverse port forwarding as well, which is very useful for certain debugging tools.

Commands

Our sample devspace.yaml includes some Laravel and MySQL-specific commands to ease development workflow.

You can run any artisan, composer, php, and npm commands and additionally drop into a MySQL shell with a single mysql command.

You can list available commands via:

$ devspace list commands

 Name           Command                                                            Description
 artisan        devspace enter -c php -- php artisan                               Entry point for artisan commands.
 composer       devspace enter -c php -- composer                                  Entry point for composer commands.
 php            devspace enter -c php -- php                                       Entry point for PHP commands.
 npm            devspace enter -c php -- npm                                       Entry point for NPM commands.
 generate-key   TMP_FILE=.devspace/app_key.tmp && docker run --rm -v $PWD:/ap...   Generate APP_KEY.
 mysql          devspace enter -c mysql -- mysql -h'mysql' -P'3306' -u'larave...   Enter to MySQL shell.

Try these commands to get familiar with them in your workflow.

Hooks

Hooks are a quite valuable feature of DevSpace. With hooks, you can run commands before and after certain deployments.

We have defined several hooks in the devspace.yaml file, such as:

Changing MySQL user and password
Running npm run watch on the PHP container.
Reloading Nginx to re-read the configuration.

Deploying to Production

The devspace deploy command will deploy our application into the environment we define. DevSpace configuration allows us to modify and alter our parameters based on profiles. This flexibility brings numerous configuration options for development, staging, production environments. DevSpace configuration can hold all different parameters and configurations for each environment. Generally speaking, it is a good practice to create a production profile for deployment, which will remove troubleshooting aids and set parameters accordingly.

Our prepared devspace.yaml consists of a production profile that will remove the additions we have made to make developing and troubleshooting easy.

Deploy with production profile:

$ devspace deploy -p production

Troubleshooting with DevSpace

Troubleshooting and debugging are pretty straightforward with DevSpace. DevSpace provides aid for the following:

Logging
Entering into containers
Running commands inside the containers
Interactive mode

Entering to and Working with Containers

The devspace enter command allows you to open a shell to any of the running containers by providing the container name, so you don't have to deal with the copy/paste of long pod names.

$ devspace enter -c php

[info]   Using namespace 'default'
[info]   Using kube context 'docker-desktop'
[info]   Opening shell to pod:container app-0:php
root@app-0:/var/www/html#

If a container is not specified, a selector will be displayed, and you can choose from the available containers:

? Which pod do you want to open the terminal for?
  [Use arrows to move, type to filter]
> redis-master-0:redis  app-0:nginx
  app-0:php
  mysql-0:mysql

Testing

As you can execute any command in the container, running the tests you have for the application is a breeze.

You can easily run phpunit or artisan test commands for running your tests.

$ devspace run php ./vendor/bin/phpunit

[info]   Using namespace 'default'
[info]   Using kube context 'docker-desktop'
[info]   Opening shell to pod:container app-0:php
PHPUnit 9.5.3 by Sebastian Bergmann and contributors.

.F                                                                  2 / 2 (100%)

Time: 00:00.122, Memory: 20.00 MB

There was 1 failure:

1) Tests\Feature\ExampleTest::testBasicTest
Expected status code 200 but received 302.
Failed asserting that 200 is identical to 302.

/var/www/html/vendor/laravel/framework/src/Illuminate/Testing/TestResponse.php:187
/var/www/html/tests/Feature/ExampleTest.php:19

FAILURES!
Tests: 2, Assertions: 2, Failures: 1.

Let's run artisan test:

$ devspace run artisan test

[info]   Using namespace 'default'
[info]   Using kube context 'docker-desktop'
[info]   Opening shell to pod:container app-0:php

   PASS  Tests\Unit\ExampleTest
  ✓ basic test

   FAIL  Tests\Feature\ExampleTest
  ⨯ basic test

  ---

  • Tests\Feature\ExampleTest > basic test
  Expected status code 200 but received 302.
  Failed asserting that 200 is identical to 302.

  at tests/Feature/ExampleTest.php:19
     15▕     public function testBasicTest()
     16▕     {
     17▕         $response = $this->get('/');
     18▕ 
  ➜  19▕         $response->assertStatus(200);
     20▕     }
     21▕ }
     22▕ 


  Tests:  1 failed, 1 passed
  Time:   0.19s

CI/CD

DevSpace configuration can hold many profiles and can be used for different deployment options. It is common to see developers use DevSpace for their CI/CD pipeline as well. Deploying your application into the CI/CD pipeline is relatively straightforward. The ability to choose from various profiles makes it a breeze to switch from development to staging and production.

Clean Up

You can easily clean your environment with the devspace purge\ command. This will be deleting all deployments. Please note that purge will not delete persistent storage(s).

Conclusion

We have seen DevSpace in action while developing, deploying, and troubleshooting. We have seen that once DevSpace is configured, it can encompass all deployment options within it. So it can be used for development and deployment to any environment. The ability to change profiles, add new commands, and execute any hooks is advantageous.

The second part of this series will delve into how to configure DevSpace, and we will go over the many possible configuration options.

Kubernetes Monitoring Dashboards - 5 Best Open-Source Tools

Rich Burroughs — Tue, 17 Aug 2021 16:40:28 +0000

by Tyler Charbonneau

Kubernetes now runs in more than 70 percent of container environments. Monitoring has become a key way to extract as much information as possible during container runtime. This data is critical when troubleshooting issues. It’s also integral to optimizing performance, both proactively and reactively.

However, Kubernetes presents a unique challenge on two fronts: setup and monitoring. To begin, it’s difficult to really nail your deployment in an organized, high-performing way. Common mistakes involve incorrectly sizing your nodes, consolidating containers, or properly creating namespaces. Making resource requests via a configuration file or kubectl requires strong forethought.

Consider this: roughly 49 percent of containers use under 30 percent of their requested CPU allocation, and 45 percent of containers use less than 30 percent of their allotted memory. Real-time monitoring can help prevent these problems. Idle resources are expensive and don’t provide any real benefit to your ecosystem.

Metrics Tracking

These missteps can require mitigation sooner or later. Accordingly, reliably tracking runtime metrics like latency, CPU utilization, and memory usage is often tricky. Other important metrics include the following:

Cluster state metrics—like pod health and availability
Node status—including readiness status, CPU/memory/disk load, and network status
Pod availability
Disk utilization

Kubernetes doesn’t always excel at displaying this data in a meaningful and readable way. It’s up to you—the DevOps professional—to piece together bespoke solutions. Designing a custom dashboard is difficult and time-consuming. Thankfully, a number of third-party vendors have created capable visualization tools for Kubernetes. Because these tools are open source, they also interface effectively with some adjacent technologies.

Want to keep your Kubernetes deployment healthy and running? Follow along to learn more about these best open-source Kubernetes dashboards.

Evaluation Criteria

How do you judge what makes a tool favorable? For analysis purposes, we’ll primarily dive into these assessment categories:

Metrics availability
Usability
Setup and maintenance requirements

These comparisons are more nuanced than just comparing hard numbers. It’s important to look at standout features and any features that distinguish one tool from the next. Each solution’s “secret sauce,” as it were, could be uniquely beneficial for your custom deployment. There are also many subjective ways to measure a tool’s worth. Quality of documentation, information presentation, and even graphical user interface (GUI) differences can shape tooling opinions.

This guide focuses predominantly on objective aspects but will introduce other notable characteristics that might sway your decision. Balancing functional needs with personal preferences is critical. Here are some top picks on both the server and client sides:

Server-Side Tools

Many teams might opt for server-side monitoring tools to capture Kubernetes data. Kubernetes natively captures resource utilization data and aggregates it in a database. This is known as the resource metrics pipeline. Both the Horizontal Pod Autoscaler controller and the kubectl-top utility generate this data during usage—which Kubernetes temporarily collects via an in-memory metrics-server. This information is exposed through the metrics.k8s.io API, which allows external services to tap into usage data.

The metrics-server requests all resource metrics from discovered cluster nodes via kubelet. Furthermore, kubelet will dig deeper by translating pods into associated containers—ultimately exposing that information with the Resource Metrics API.

Additionally, DevOps teams can leverage the full metrics pipeline to view more intricate data. This also taps into nodes via kubelet, but either the custom.metrics.k8s.io or external.metrics.k8s.io APIs expose that information instead. Note that Kubernetes can react natively based on these gathered metrics to counteract problems. The onus isn’t entirely on the development side.

Server-based metrics tracking has some significant advantages. This collection method is reliable due to its simplified collection approach; you don’t have to wrestle with thousands of nodes, pods, or containers individually. Because server tracking offloads the burden from your Kubernetes infrastructure, you’ll also see a performance improvement. It’s relatively easy to wrangle your data as required from the server once it’s up and running.

However, no system is perfect, and this fact also applies to server-side monitoring. These solutions can be harder to configure, as you have to install essential components within your system that can effectively transmit data elsewhere. There can be more failure points across the system. Server-side transitions can also be costlier. The overall breadth of the server-side data you collect might be more limited—or at least be missing some crucial insights of interest. Server logs aren’t always human-readable, and logs might only be retained for a limited time unless they’re archived.

That said, millions of users have come to love their server-side monitoring tools. Here are some of the top dogs in the category that have become household names:

Kubernetes Dashboard

Image courtesy of Chuka Ofili.

Offered natively through the web browser, the Kubernetes Dashboard operates as a web app, offering detailed insight into your containerized environment. While some open-source solutions are read-only, the dashboard allows you to deploy, troubleshoot, and actively manage system components. The tool excels at managing both system resources (Jobs, DaemonSets, deployments) and applications. It’s essential to monitor both the system and the microservices that run atop it.

You can also monitor the following via the Dashboard, all accessed easily via the left-hand sidebar:

Namespaces
Nodes
Persistent volumes
Roles
Storage classes
Cron jobs
Replica sets and replication controllers
Stateful sets
Discovery and load balancing parameters
Configurations and storage setups

Each of these categories has a detailed view with easy-to-read infographics. Because the Kubernetes Dashboard is web-based, you can access it anywhere at any time. This makes it a fantastic, platform-agnostic tool that functions admirably regardless of one’s operating system—or even cluster architecture. To deploy a containerized application, simply connect your YAML or JSON configuration file with the integrated setup wizard.

On the logging side, the built-in Logs Viewer pulls records from containers belonging to single pods and displays them in a list format. That output is organized in chronological order for easier parsing. You can download these specific logs as needed with a single click.

The Dashboard isn’t installed by default. Run the following command to deploy it:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml

You can access Dashboard via kubectl by running the kubectl proxy command. Only machines having executed this command can view the Dashboard UI. Additionally, your deployment will adhere to simple role-based access control (RBAC) standards, exclusively necessitating a Bearer token for successful login.

Skooner (Formerly K8dash)

Having received a face-lift and new name, Skooner continues to be a leading open-source tool for holistically monitoring Kubernetes. The developers behind the project tout the simplicity and real-time availability of their solution—no refreshes or manual polling is required to fetch system data as it’s collected. Additionally, the YAML provided within the tool’s resource repository allows you to start leveraging Skooner in just a minute’s time.

There’s very little of a learning curve involved on the setup side. All you need is a running Kubernetes cluster, a recommended metrics-server installed, and an optional OpenID Connect configuration. Unlike with the Kubernetes Dashboard, you can log into Skooner using one of three methods: a service account token, OpenID Connect (OIDC), or via NodePort. The first is the easiest while the last is the fastest, per the developers. Those favoring OpenID will naturally gravitate toward OIDC. Note that kube proxy cannot be used to access this Dashboard, as the Authorization header is stripped during execution.

Here’s what you can visualize using Skooner:

Namespaces
Nodes
Pods
Replica sets
Deployments
Storage
RBAC configurations
Workloads

Skooner relies heavily on metrics-server to pull runtime metrics. Without this component installed, the platform’s functionality will suffer to a degree. It’ll be much more difficult to summon utilization data, for instance, without tapping into that pipeline.

However, Skooner brings its mobile app to the table. It runs on most phones or tablets, allowing you to keep tabs on crucial metrics while you’re on the go. The solution is highly scalable—responding well to changes in Kubernetes system configurations and growth while continuing to grab relevant information reliably.

Prometheus UI

Image courtesy of Christiaan Vermeulen.

Last but not least, Prometheus is a wildly popular open-source tool maintained by the Cloud Native Computing Foundation (CNCF). As such, it enjoys significant backing and development support by the community at large. You’ll notice an immediate GUI difference between this and the other entrants on our list; Prometheus uses a darker palette and arranges its visualizations a little differently. The information is displayed more densely according to endpoint, host, and port. Prometheus helps monitor the following:

CPU utilization (including core counts)
RAM usage (including total available)
SWAP memory usage (from total)
Root filesystem usage (from total)
CPU system load (per interval average)
Uptime

Prometheus stores data as a time series, where metrics have streams of time-stamped values. Each metric also has its own set of labeled dimensions. These are typically stored, though Prometheus can generate temporary series from proprietary PromQL queries. Visually, metrics are commonly displayed as graphs and the like by linking Prometheus with Grafana—which pulls from an assigned data source.

Getting started with Prometheus requires you to install its exporter component on each relevant Kubernetes node. This acts as a service that streams runtime data to the database and dashboard later on. You can set up a multidimensional data model using queries and key-value pairings.

You can install Prometheus using a pre-compiled binary, via Docker images and volumes or configuration management systems like Chef or Ansible. Prometheus is also good at self-monitoring through the use of included APIs. Finally, Prometheus’ massive community can provide help with this task or any other facet of the monitoring process.

Client-Only Tools

As opposed to server-based monitoring, client-only tools are best for teams needing an easy solution without excessive configuration. They’re generally cheaper and have lower barriers to entry. Those running stateless applications in particular might benefit from client-side monitoring, as critical session or resource data isn’t typically stored on the server. Here are our picks:

Lens by Mirantis

As an ops-focused monitoring tool, Lens is a popular Kubernetes integrated development environment (IDE) that acts as a multidisciplinary continuous integration/continuous delivery (CI/CD) platform. The service bundles a contextual terminal with Prometheus-derived statistics while ensuring that logs are easily viewable. Monitored clusters may either be local or external. Accordingly, you can even add a cluster into the mix by importing a kubeconfig file.

Clusters and their tracked metrics are separated into working groups—useful for different teams or for maintaining segregation within complex deployments. You can summon real-time graphs within the Lens dashboard that are tailored to each namespace and resource. Thanks to included RBAC controls, you can define which users can access specific metrics for greater security. Lens remains open source and free to use.

Octant by VMWare

Like the Kubernetes Dashboard, Octant is an open-source web interface for visualizing your clusters and applications. The solution supports multiple plugins via a core gRPC API, making Octant extensible and richly featured. Like other tools, it provides real-time updates on the health and performance of your cluster’s objects plus related objects. This detailed metrics tracking is meant to simplify the debugging process and highlight problems before they become threatening. Building off of kubectl and kustomize, Octant is a simple and reliable tool for managing the Kubernetes system as a whole.

Conclusion

We work in a fast-paced industry, and the tools available for Kubernetes can change quickly. No matter what, your choice of tools will chiefly depend on your needs and the unique deployment you’ve built with Kubernetes. You’ll want to consider your team’s experience and comfort level when choosing an approach—either server-based or client-based. Open-source tools for monitoring Kubernetes are thankfully robust and widespread across the marketplace. If you harness real-time information, keeping your Kubernetes deployment healthy and operative should be much simpler.

Photo by William Warby on Unsplash

Save Costs With Virtual Kubernetes Clusters

Rich Burroughs — Fri, 30 Jul 2021 17:21:31 +0000

by Fabian Kramm

Virtual Kubernetes clusters are fully functional Kubernetes clusters that run within another Kubernetes cluster. The difference between a regular Kubernetes namespace and a virtual cluster is that a virtual cluster has its own separate Kubernetes control plane and storage backend. Only a handful of core resources, such as pods and services, are actually shared among the virtual and host cluster. All other resources, such as CRDs, statefulsets, deployments, webhooks, jobs, etc., only exist in the pure virtual Kubernetes cluster.

This provides a lot better isolation than a regular Kubernetes namespace and decreases the pressure on the host Kubernetes cluster as API requests to the virtual Kubernetes cluster in most cases do not reach the host cluster at all. In addition, all created resources by the virtual cluster are also tied to a single namespace in the host cluster, no matter in which virtual cluster namespace you create those resources in.

With version v0.3.0, vcluster an open source implementation of the virtual Kubernetes cluster pattern and that builds upon the lightweight Kubernetes distribution k3s, now also became a certified Kubernetes distribution and is 100% Kubernetes API compatible. This makes virtual clusters now even more interesting to use.

How can Virtual Kubernetes clusters decrease costs?

In essence, virtual Kubernetes clusters are a trade-off between namespaces and separate Kubernetes clusters. They are easier and cheaper to create than fully blown clusters, but they are not as well isolated as completely separate clusters, since they still interact with the host Kubernetes cluster and create the actual workloads in it. On the other hand, they provide much better isolation than namespaces. Virtual clusters use a completely separate control plane, and within a virtual cluster you have full cluster-wide control access. Nonetheless, a single namespace is still cheaper and easier to create.

The table below summarizes the differences between Namespaces, virtual Kubernetes clusters, and fully separate clusters.

The important takeaway from this is that virtual Kubernetes clusters provide a new alternative to both namespaces and separate clusters. Virtual Kubernetes clusters provide an excellent opportunity to replace separate clusters and drastically reduce your infrastructure and management costs, especially in scenarios where you have at least basic trust in your tenants (say separate teams across your company, CI/CD pipelines, or even several trusted customers).

An Example Scenario

Let's say you are a company that provides some sort of SaaS service, and you have around 100 developers distributed across 20 teams that implement different parts of the service. For each of those 20 teams, you provisioned separate Kubernetes clusters to test and develop the application, as this was the easiest and most flexible approach. Each team’s cluster has at least three nodes to guarantee availability and then automatically scales up and down based on usage.

Your minimum infrastructure bill in Google Cloud might look like this over 12 months (according to the Google Cloud Pricing Calculator):

Node Cost:
20 Clusters * 3 Nodes (n1-standard-1) = 12 * 20 * $72.82 = $17,476.8

GKE Management Cost:
20 Clusters (Zonal) = 12 * 20 * $71.60 = $17,184

Total Cost Per Year (Without Traffic etc.):
$17,476.80 + $17,184 = $34,660.80

In total, you are looking at a minimum estimated raw node + management cost of about $35,000. Obviously, you could still fine-tune certain aspects here, for example reducing the minimum node pool size or using preemptive nodes instead of regular nodes.

The advantages of this setup are clear. Each team has its own separate Kubernetes cluster and endpoint to work with and can install cluster-wide resources and dependencies (such as a custom service-mesh, ingress controller, or monitoring solution). On the other hand, you'll also notice that the cost is quite high. Resource sharing across teams is rather difficult, and there is a huge overhead if certain clusters are not used at all.

Switching to virtual Kubernetes clusters

This is a perfect example where virtual Kubernetes clusters could come in handy. Instead of 20 different GKE clusters, you would create a single GKE Kubernetes cluster and then deploy 20 virtual Kubernetes clusters within it. Now each team gets access to only a single virtual Kubernetes cluster endpoint that essentially maps to a single namespace in the underlying GKE cluster.

The really great part about this is that from the developers’ perspective, nothing has changed. Each team can still create all the cluster services they want within their own virtual cluster, such as deploy their own Istio service mesh, custom cert-manager version, Prometheus stack, Kafka operator, etc. without affecting the host cluster services. They can essentially use it the same way as they would have used the separate cluster before.

Another benefit is that the setup is now much more resource-efficient. Since the virtual Kubernetes clusters and all of their workloads are also just simple pods in the host GKE cluster, you can leverage the full power of the Kubernetes scheduler. So, for example, if a team is on vacation or is not using the virtual Kubernetes cluster at all, there will be no pods scheduled in the host cluster consuming any resources. In general, this means the node resource utilization of the GKE cluster should now be much better than before.

Another significant advantage with a single GKE cluster and multiple virtual clusters in it is that you as the infrastructure team can centralize certain services in the host cluster, such as a central ingress controller, service mesh, metrics, or logging solutions instead of installing it every time into all of the separate clusters. The virtual clusters will be able to consume those services, or if the teams prefer they can still add their own. Teams will also be able to access each other’s services if that is needed, which would be very difficult with completely separate clusters.

Furthermore, you will save on the cloud provider Kubernetes management fees by sharing resources better. And vcluster is open source, so it’s self-managed and completely free. The new cost estimate would look a little bit more like this if you would reserve a node for each team and add three extra nodes as a high availability buffer:

Node Cost:
1 Clusters * 23 Nodes (n1-standard-1) = 12 * $558.26 = $6699.12

GKE Management Cost:
1 Clusters (Zonal) = 12 * 1 * $71.60 = $859.20

Total Cost Per Year (Without Traffic etc.):
$6699.12 + $859.2 = $7558.32

Total Cost Savings:
$34,660.80 - $7558.32 = $27,102.48 (78.2% savings)

In this case, your minimum node & GKE management fee infrastructure bill would be cut down by 78.2%. This is a rather constructed example, but it shows the considerable potential of virtual clusters. For the teams using the virtual clusters, essentially nothing would change because they would still have access to a fully functional Kubernetes cluster where they could deploy their workloads and cluster services freely.

Conclusion

Virtual Kubernetes clusters are a third option if you have to decide between namespaces or separate clusters. Virtual clusters will probably never completely replace the need for separate clusters. Still, they have significant advantages if your use case fits, as you can save significant infrastructure and management costs with them.

Photo by Kelly Sikkema on Unsplash

Let's Learn vcluster with Saiyam Pathak

Rich Burroughs — Tue, 13 Jul 2021 17:06:30 +0000

Loft Labs CEO Lukas Gentele joined Saiyam Pathak on his stream to talk about vcluster. This video is another great way to get up to speed on vlcuster if you've heard about it and are curious. In the video, Lukas walks Saiyam through the different vcluster features, including the workflow, how to use it with Ingress, and storage considerations.

Saiyam creates a ton of great content about Kubernetes. Check out the other videos on his YouTube. He's also hosting a show on Cloudnative.tv about Kubernetes certifications called Cert Magic, and he's written a book about the scenarios in the CKS exam.

Docker Compose vs Kubernetes Development Tools

Rich Burroughs — Mon, 12 Jul 2021 22:32:46 +0000

By Kasper Siig

When getting started with Docker, many developers quickly turn to Docker Compose to run their applications. Compose offers many advantages, such as having your configuration stored as code, making it easy to maintain and expand upon. Unfortunately, although it is possible to use Compose with Kubernetes, it's not the recommended approach.

Devs will often bang their head against the wall trying to make this scenario work when they start using Kubernetes, without knowing that there's a better way. After all, they have become used to Compose and have integrated it deeply into their workflow. It can be hard to let go.

This article will go over why it's best to leave Compose out of Kubernetes, and give resources to help you with improving your workflow without it. You'll be introduced to tools that will provide you with the same advantages as you would have with Compose traditionally.

Avoiding Docker Compose in Kubernetes

So you've started to use Kubernetes with your project, and you're wondering what to do with all of your work in Docker Compose. Rather than having to abandon all of your work and start completely from scratch, it's possible to use a tool like Kompose as a way of converting your docker-compose.yml files into Kubernetes manifests. Being familiar with Compose, this can give you great insights into how things are mapped into Kubernetes, and act as a starting point for your research into manifests.

However, when you start moving from the learning phase to the production phase, it's important to think about whether you want to keep Compose in your toolchain at all. Even though tools like Kompose exist to help bring Compose into a Kubernetes environment, it's still not considered best practice. Instead, you should consider switching over to using Kubernetes manifests.

Using Compose in production can be fine initially, and if your goal is to get simple containers deployed, it's not a big deal. That being said, once your cluster starts maturing and your use cases become more complex, you will find that trying to define everything in a docker-compose.yml is either tough or impossible.

You'll likely get to a point where you're spending a significant amount of time developing and maintaining docker-compose.yml files. So much so that it would have been easier to just start over with Kubernetes manifests. This is an important point to consider when deciding whether to use one tool over another. One may be easier initially but perhaps limits the possibilities in the future, or be harder to work with in complex scenarios.

Compose's Consequences and Risks

By using Compose in Kubernetes, you are limited in functionality. While Compose is a robust tool with a rich feature library, there are many things it cannot do. Objects like CRDs, Jobs, and StatefulSets cannot be created with Compose. Networking is possible, but it can quickly become unwieldy to define it in a docker-compose.yml file.

There are some technical downsides to the continuing use of Compose, but you will have to also consider the impact on your team, both current and future. Not many people are using Compose in production, so you'll likely struggle to find a new hire that's able to jump right in. There are also features of Compose that are not typically used, which you'll have to get familiar with to configure Kubernetes.

If you manage to get your engineers to learn and use Compose efficiently, and you're fine with onboarding new people into the toolchain, you may still run into issues. Since not many teams use Compose in production, it can be tough to find guides and tutorials with examples. Many online resources will only include Kubernetes manifests as examples, and from here two things can happen:

One option is that the engineer in the team will understand the tutorial and get everything defined in a .yml file. This way, you'll continue to purely use Compose, but you'll have to carry the cost of engineering time spent converting the Kubernetes manifest. This also means that your engineers understand manifests well enough to convert them to another format, weakening the argument for using Compose.
The other option is that the example manifest will be used as a Proof of Concept, but it will end up being used in production because of a deadline or other reasons. Now you have a mix of Compose files and Kubernetes manifests, which can quickly lead to confusion.

You will have a tough time integrating with other tools on the market since many tools exist to expand upon existing Kubernetes manifests. Some of these tools help in easing deployment, like Helm. Other tools like Skaffold work with your manifests to run your application in Kubernetes as you work on it. You might find workarounds that allow you to use these tools, but you won't find any official documentation on setting them up. You'll have to maintain these workarounds, and it creates more room for error.

Finally, you run the risk of having different teams using different tools. Developers may want to use Compose as it's more user-friendly on the surface, and they mostly care about getting the application to run and making optimizations through the code. Ops may want to get deeper into the roots of Kubernetes in ways only possible when using native Kubernetes tools. Typically they care about optimizations in the infrastructure, like networking and load balancing. Using Kubernetes manifests won't guarantee different teams using the exact same tools, but they will have the same common ground.

Other Options

As stated before, there are many tools available to help you work with Kubernetes. Many stick with Compose because it's easy to define containers, networking, and volumes, and that's a fair point. However, tools like Skaffold, DevSpace, and Tilt exist to make working on code that's meant to run in Kubernetes easier. These tools offer features such as watching your code, automatically building and deploying your application, and much more that's native to Kubernetes.

These tools can help you transition from a Compose-based approach into something more akin to native Kubernetes. Their sole goal is to make life easier for developers while still using the basis of Kubernetes; manifests. Give them a try and see how they work for you, and whether you can find a way of getting them into your current toolchain. To get started, you can use Kompose as a way of converting your existing docker-compose.yaml files into Kubernetes manifests. From here, you can either deploy them and get familiar with the deployment process, or you can look into the generated files and try to understand them.

Whatever tool you choose to go with, the most important thing is that you know why you're using it. Many best practices exist because they're what suits most organizations. However, there will always be outliers, and you may be one of them. You may be in a situation where it indeed does make the most sense to use Compose as your only tool, and that is perfectly acceptable.

You just need to know why you've chosen to go with the tool that you have, making it possible to reevaluate down the line whether it's still right for you or if you should consider switching to best practices.

Conclusion

On the surface, it can seem challenging to learn a new tool, and Kubernetes is quite heavy for new users. However, transitioning from Compose to native Kubernetes isn't as complicated as it may seem, and as you've now seen, there are many tools available to assist you with this. Switching to manifests will help you in many ways. Whether you make the switch is up to you, but consider whether it's the right choice and what advantages it can bring to you.

You can start by converting your docker-compose.yml to Kubernetes manifests with Kompose. That way you'll be using an application and definition that you're already familiar with instead of starting from scratch with an application you don't know.

Photo by Campbell on Unsplash

DEV Community: Rich Burroughs

Cyclops: Platform Engineering for the Rest of Us

Prerequisites

Provision a cluster and install Cyclops

Deploy Niginx with a generic app template

Deploy Redis with Cyclops

Conclusion

Learn more

KubeCon Chicago Wrapup

Pre-Conference Activities

Day One - Tuesday

Keynotes

A Practical Guide to eBPF Licensing: Or How I Learned to Stop Worrying and Love the GPL - Jef Spaleta & Bill Mulligan, Isovalent

When Is a Secure Connection Not Encrypted? and Other Stories - Liz Rice, Isovalent

Demystifying Service Mesh: Separating Hype from Practicality - Brian Redmond & Ally Ford, Microsoft

Service Mesh Battle Scars: Technology, Timing, and Tradeoffs - Keith Mattix, Microsoft; John Howard, Google; Lin Sun, solo.io; Thomas Graf, Isovalent; Flynn, Buoyant

Day Two - Wednesday

Keynotes

Learning Kubernetes by Chaos – Breaking a Kubernetes Cluster to Understand the Components - Ricardo Katz, VMware & Anderson Duboc, Google Cloud

Dungeons and Deployments: Leveling up in Kubernetes - Noah Abrahams, Oracle; Natali Vlatko, Cisco; Kat Cosgrove, Dell; Seth McCombs, AcuityMD

Documentary Film - eBPF: Unlocking the Kernel

Day Three - Thursday

Keynotes

Sidecar Containers Are Built-in to Kubernetes: What, How, and Why Now? - Todd Neal, Amazon & Sergey Kanzhelev, Google

Releasing Kubernetes and Beyond: Flexible and Fast Delivery of Packages - Grace Nguyen, University of Waterloo; Adolfo Garcia Veytia, Chainguard; John Anderson, Ditto

Conclusion

Managing Access to Kubernetes Clusters for Engineering Teams

Why You Need Kubernetes Clusters

Provisioning of Multiple Environments

Running Multiple Deployments

Easy Scaling of Deployments

Managing Access in Your Kubernetes Cluster

Granting Users Access

Managing Access

Providing Only Needed Permissions

Decommissioning Users as Needed

Enabling Auditing

Using Loft for Access Control

Best Practices

Conclusion

Kubernetes Cost Monitoring with Prometheus & Grafana

Setting Up Prometheus Operator

Monitoring Your Cost

Alternative Options

Kubernetes Network Policies: A Practitioner's Guide

Why We Need Network Policies

Requirements for Implementing Network Policies

Network Plugins

Writing & Applying Network Policies

Isolation

Network Policy Resource Fields

Egress Rules

Ingress Rules

Walkthrough

Examples

Default Deny Ingress

Allow Access to a Group of Pods from Another Namespace

Monitoring Network Policies

Conclusion

Further Reading

Docker Compose Alternatives for Kubernetes: DevSpace

Architecture

Installation

Initializing Your Project

Entering Development Mode

Hot Reloading For Containers

Developing Microservices with Kubernetes

Hooks and Custom Commands

Web UI For Kubernetes Development

Conclusion

Loft Feature Spotlight: Sleep Mode

How Sleep Mode Works

Enabling Sleep Mode

Manually Triggering Sleep Mode

Conclusion

PHP Laravel Development with Kubernetes using DevSpace - Developer Edition

Architecture

Introduction to DevSpace

Feature Highlights

Requirements and Setting Up Development Environment