DEV Community: Distr

Should I Run Plain Docker Compose in Production in 2026?

Philip Miglinci — Thu, 30 Apr 2026 13:13:57 +0000

I am Philip—an engineer working at Distr, which helps software and AI companies distribute their applications to self-managed environments.
Our Open Source Software Distribution platform is available on GitHub (github.com/distr-sh/distr) and orchestrates both Docker Compose and Docker Swarm deployments on customer hosts every day.

Most of the production incidents I have seen on Docker Compose hosts come from the same handful of quirks: an old container that should have been removed, a disk that filled up overnight, a health check that detected a problem and then did nothing about it, a :latest tag that pointed somewhere new, or a socket mount nobody thought twice about. None of these are bugs in Docker. They are deliberate trade-offs in a tool that started as internal tooling at dotCloud, a PaaS company that wrapped LXC to fix "it works on my machine," and is now running the back end of a lot of real businesses. This post collects the recurring ones, with the commands and the operational answer for each.

Short answer: yes—plain Docker Compose can still run real production workloads in 2026, but only if you handle the operational gaps it leaves yourself.

Where Plain Docker Compose Fits in Production

Before the list of quirks, a quick word on the audience. Docker Compose is a declarative way to wire up a multi-container application: one YAML file describes the services, the networks between them, the volumes they share, the environment they need, and—through the patterns for overwriting or patching service configuration—the on-disk configuration each application expects. docker compose up reconciles the host to that file. The sweet spot in production is the single-node deployment built around exactly that—a vendor pushing a multi-container application into a customer environment, an internal team running a long-tail service that does not justify a Kubernetes cluster, an edge box in a retail location. The footprint is small, the operational overhead is low, and a competent operator can reason about the whole stack from one docker-compose.yaml. There is no control plane behind Compose itself—no scheduler watching the host, no reconciler reapplying state, no operator pushing updates from somewhere else. docker compose up runs once and exits.

That architectural simplicity is exactly why the quirks bite. Compose assumes you—or whoever runs the host—will do the operational work nothing else is doing, and if you ship Compose files to customers the safe assumption is that the customer will not. The rest of this post is about closing the gap between what Compose does and what a production host actually needs, either by hand or with an agent that does it for you. If you have already concluded that the gap is too wide and want to compare with the next step up, read our Docker Compose vs Kubernetes breakdown.

Docker Compose Orphan Containers and `--remove-orphans`

Remove a service from docker-compose.yaml, run docker compose up -d, and the container you removed keeps running. It is detached from the project but still bound to the same networks and ports. docker compose ps will not show it, because Compose only lists what is in the current file. docker ps --filter label=com.docker.compose.project=<name> will, because Docker still has the label on the container. This is how you discover, six months in, that an old worker service has been quietly consuming RAM since the last refactor.

The fix is one flag:

docker compose up -d --remove-orphans
docker compose down --remove-orphans

The flag tells Compose: any container that was once part of this project but is no longer in the file should be removed. Networks Compose created for the project are reconciled the same way on each up, so orphan networks go away too. Volumes are the exception—Compose preserves named volumes by default to protect data, and there is no per-service flag to drop the ones a removed service used. To reclaim that space you have to do it manually: list candidates with docker volume ls --filter dangling=true and docker volume rm by name, or use docker compose down -v if you intend to wipe the project's volumes wholesale. To audit before deleting, list everything Docker still associates with the project name:

docker ps -a --filter label=com.docker.compose.project=<name>

--remove-orphans only touches containers that share the Compose project
label. It will not remove unrelated containers on the host. The risk is
bounded to containers Compose itself created earlier under this project name.

Distr's Docker agent passes RemoveOrphans: true on every Compose Up call, so customer hosts never accumulate orphans across deployment updates. That single flag has eliminated a recurring class of "the old version is still answering on port 8080" support tickets.

Pruning Docker Images and Capping Container Logs

Every docker compose pull keeps the previous image on disk. Every container with the default json-file log driver writes unbounded JSON to /var/lib/docker/containers/<id>/<id>-json.log. On a busy host this is one of the most common reasons for an outage: the disk fills and Docker stops being able to write anything—logs, metadata, image layers—at which point containers start failing in confusing ways.

The first thing to learn is the audit command:

docker system df
docker system df -v

-v breaks the totals down per image, container, volume, and build cache, which is usually enough to spot the offender. From there, the targeted prune commands:

docker image prune -a --filter "until=168h" -f   # delete unused images older than 7 days
docker container prune -f                         # remove stopped containers
docker builder prune -f                           # drop the BuildKit cache

docker volume prune -f exists too, and it is genuinely useful, but read the next aside before you run it.

docker volume prune removes every volume not currently attached to a
container, including named volumes you created on purpose. If you stop a
database container to do maintenance and run docker volume prune while it is
down, you can lose its data. Use docker volume ls --filter dangling=true
first to see what is actually unreferenced, and prefer named-volume management
over blanket prunes on production hosts.

The other half of the disk story is logs. Cap them at the daemon level, once, in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

After systemctl restart docker, every new container will rotate its logs at 10 MB and keep at most three rotated files—30 MB ceiling per container, instead of "until the disk is gone." Existing containers need to be recreated to pick up the new defaults.

This is one of the topics worth getting right before you ship.

In Distr's Docker agent the cleanup is built in: each deployment target has an opt-out container image cleanup setting that removes the previous version's images automatically after a successful update, with retries on failure. It only fires on success, so the previous image stays on disk if something goes wrong and you need to roll back.

Docker Health Checks Don't Restart Unhealthy Containers

This is the one that surprises people the most. You add a HEALTHCHECK to your Dockerfile or a healthcheck: block to the service in Compose, you watch the container go from healthy to unhealthy, and then... nothing happens. The Docker Engine reports the status. It does not act on it. restart: unless-stopped is triggered by the container exiting, not by it being marked unhealthy.

You can confirm what Docker actually thinks:

docker inspect --format='{{json .State.Health}}' <container> | jq

You will see the status, the streak of failures, and the last few probe outputs—useful information that is silently ignored by the engine.

There are three answers to this:

Run an autoheal sidecar. The community standard is willfarrell/docker-autoheal: a tiny container that mounts the Docker socket, watches for unhealthy events, and restarts the offending container. You opt containers in by labeling them autoheal=true (or set AUTOHEAL_CONTAINER_LABEL=all to monitor everything).
Run on Docker Swarm. Swarm restarts unhealthy tasks by default. If you are already considering Swarm, this is one of the better reasons.
Use Distr. Every Distr Docker agent deploys an adapted autoheal service alongside it. The "Enable autoheal for all containers" toggle is on by default at deployment-target creation, so customer-side restarts of unhealthy containers happen without anyone configuring it.

In Distr the autoheal setting is fixed at deployment-target creation and
cannot be toggled later. If you need to change it on an existing target,
recreate the target. Worth knowing before you provision.

Whichever path you pick, the takeaway is the same: a HEALTHCHECK without something acting on it is a status light, not a self-healing system.

Pinning Docker Images by Digest Instead of `:latest`

Docker tags are mutable references. myapp:1.4 today is whatever the registry currently has under that tag; tomorrow it can point at a different layer set after a re-push. :latest is the worst offender because everyone treats it as a synonym for "stable" when in practice it often means "whatever was pushed most recently." It is also the silent default: an unqualified image: nginx in a Compose file is treated as image: nginx:latest, so even Compose files that never type the word land on it by accident. The result, in production, is that two hosts pulling the "same" tag five minutes apart can end up running different code.

The fix is to pin by content-addressable digest. Every image has one, and Docker accepts it anywhere a tag would go.

To find the digest for an image you already pulled:

docker image inspect --format='{{index .RepoDigests 0}}' myapp:1.4
# myapp@sha256:9b7c…

Or, without pulling, from the local Docker installation against the remote registry:

docker buildx imagetools inspect myapp:1.4

In your Compose file, replace the tag with the digest:

services:
  app:
    image: myapp@sha256:9b7c0a3e1f...

A pull against a digest fails fast if the registry no longer has those bytes, which is exactly what you want—silent drift becomes a loud error. The same image reference works in docker stack deploy, in docker run, and in Kubernetes manifests.

For the broader picture of what your customers can extract from a published image (and why image hygiene matters beyond reproducibility), check out our guide on protecting source code and IP in Docker and Kubernetes deployments. And if you're still picking a registry, our container registry comparison walks through the trade-offs.

Why Mounting `/var/run/docker.sock` Is a Security Risk

A container with /var/run/docker.sock mounted can call the Docker API, and the Docker API can launch a privileged container that mounts the host's root filesystem. In other words: any container with the socket has effectively root privileges on the host. This is not a Docker bug; it is the threat model of the socket. It deserves a moment of attention because the line that grants this access is one bind mount in a Compose file and is easy to add without thinking about it.

Mounting /var/run/docker.sock into a container is equivalent to giving that
container root privileges on the host. A :ro flag does not help—the
socket is bidirectional RPC, not a file you read. Treat every container that
mounts the socket the same way you would treat root SSH access to the host.

Practical hygiene:

Inventory the containers that mount the socket. Agents, CI runners, monitoring sidecars, container management UIs—keep the list short and intentional.
Run rootless Docker where possible. dockerd-rootless-setuptool.sh install sets up a Docker daemon that runs as a regular user. The blast radius of a compromised socket-mounting container shrinks from "full host" to "this user account."
Consider socket-proxy. Projects like Tecnativa's docker-socket-proxy expose a filtered subset of the API to the container that needs it (e.g. read-only containers and events for monitoring) instead of the full socket.
Keep socket-mounting images minimal. Smaller surface, fewer libraries, fewer ways in.

The Distr Docker agent does mount the socket—it has to, in order to orchestrate Compose and Swarm on the host. We document that boundary openly in the Docker agent docs so customer security teams can review it before installation. The agent authenticates to the Hub with a JWT, and the install secret is shown once and never stored.

Updating Docker Compose Deployments Across Customer Hosts

docker compose pull && docker compose up -d is a fine command if you are SSH'd into the host. At customer scale—dozens of self-managed environments behind firewalls, each with its own change-control process—that manual process doesn't scale. Docker has no built-in mechanism to push a new manifest to a running host from somewhere else. Docker Hub webhooks can trigger a CI rebuild when an image is pushed, but they do not reach into a customer's network and tell their docker compose to pull.

The usual workarounds and what they cost:

Watchtower: Polls the registry on a schedule, pulls new images, recreates containers. Easy to set up, hard to control. No staged rollout, no rollback path, limited visibility from your side—you find out a customer updated when they file a ticket.
Bastion + SSH + Ansible/scripts: Works for ten customers. Falls apart at fifty, especially when three of them are air-gapped and four run their own change-control cadence. Every operator has to live with shared keys and a maintenance window calendar.
A pull-based agent. This is the shape Distr lands on. The agent runs on the customer host, polls a known endpoint every 5 seconds, and reconciles the local Compose state against what the Hub says it should be. The agent reports status back, so you can see in your dashboard which customers are on which version. When the agent itself needs to update, it spawns a separate container to perform the swap so it is not trying to replace itself while running.

The pattern is not unique—Kubernetes operators and GitOps tools do the same thing—but Compose users routinely re-invent it badly. If you find yourself building one, at least give it rollback, status reporting, and a way to pin versions, or you will end up with a fleet that drifts in ways you cannot see.

The other thing worth noting: recurring scheduled jobs alongside the application have no native Compose answer either. If your stack includes anything like a nightly cleanup, a periodic report, or a heartbeat-style task, the in-app scheduler is one option, but you eventually run into the cases it can't cover (cross-service jobs, jobs that should outlive a single container). For the three patterns I have seen survive customer deployments, check out our guide on Compose cron jobs.

Outgrowing Docker Compose: Kubernetes vs Swarm

If a single-node Compose deployment outgrows itself, the realistic next step for most teams is Kubernetes. The ecosystem is large, the operational patterns are well documented, and the talent pool to hire against actually exists. For the side-by-side, read our Docker Compose vs Kubernetes comparison.

Docker Swarm is the other option—it reuses the Compose YAML format, ships in the box, and solves a few of the quirks above directly (it restarts unhealthy tasks, rolls out updates with update_config, and treats secrets and configs as first-class objects). It is a real fit for some single-cluster, low-ceremony deployments.

Swarm is not an industry standard. Adoption has plateaued, third-party tooling
is thin, and most cloud providers have moved their managed offerings to
Kubernetes. It can still be the right call for a self-contained on-prem
cluster, but pick it eyes-open—for most teams outgrowing Compose, Kubernetes
is the safer long-term bet.

The Distr agent supports both—the Hub records whether a deployment is Compose or Swarm, and the agent runs the matching docker compose up or docker stack deploy. If you do choose Swarm, read our routing and Traefik guide for Docker Swarm and the product walkthrough for distributing applications to Swarm for the details.

So, should you run plain Docker Compose in production?

Yes—plain Docker Compose still runs a lot of real production workloads in 2026, as long as you accept that "plain Compose" is shorthand for "Compose plus the operator practices it doesn't enforce." None of the quirks above are secret. They are all in Docker's documentation, in GitHub issues that have been open for years, and in the war stories of every team that has run Compose in anger. What makes them dangerous is not the quirks themselves but the order in which you discover them: usually at 2 a.m., one at a time.

TL;DR:

Pass --remove-orphans on every compose up and compose down.
Cap container logs in daemon.json and prune images on a schedule. Be careful with docker volume prune.
Health checks do not heal. Run an autoheal sidecar, run on Swarm, or use an agent that bundles one.
Pin by @sha256:... digest. Treat tags as references, not contracts.
The socket is root. Inventory the containers that mount it; prefer rootless Docker.
Updates need an agent of some kind. Watchtower is fine for one host; not for a fleet.
When Compose stops being enough, Kubernetes is usually the right next step. Swarm is a narrower fit and worth picking eyes-open.

If you ship software to self-managed customers and you would rather not rebuild this list yourself, the Distr Docker agent handles all of the above on the customer side. The Docker agent documentation walks through the install, the socket model, the autoheal and image-cleanup defaults, and how the agent self-updates. The repository is on GitHub.

Alerts for self-hosted customer deployments

Louis Weston — Tue, 17 Feb 2026 21:12:02 +0000

TL;DR: Distr is an Open Source platform to distribute applications to on-prem, BYOC, and air-gapped customers. It now supports alerts so you get notified about failed updates, crash loops, or unreachable deployments before your customers do.

The Visibility Gap

For your own application, you’ve got this figured out. Grafana alerts, PagerDuty, whatever. Something breaks in production, you know about it in minutes.

For self-hosted customer deployments? Nothing. Radio silence unless they open a ticket.

You have all this sophisticated monitoring for your own infrastructure, but the moment your software crosses over to customer-controlled environments, you’re flying blind. You find out things are broken when your customer tells you they’re broken. Sometimes days later.

You can’t SSH into every customer deployment and tail logs. That doesn’t scale, and frankly, your customers probably wouldn’t let you even if you wanted to.

What We Built

Alerts in Distr work for both sides:

Distr lets you configure alerts in the Vendor portal. Set them once, they apply everywhere. Or get granular if you want.

For customers that manage and update the deployments themselves, you can enable alerts for them as well so they can configure their own alerts in the Customer portal.

This dual approach matters because not all customer relationships look the same. Some of your customers want you monitoring their deployments—they’d rather you catch issues first and fix them proactively. Others prefer to run their own ops and want full control over who gets notified when.

For example, you’ll get alerted if an update fails or gets stuck looping, if we lose contact with a deployment (network issues, dead instance, whatever), or if your app’s in a crash loop. We filter out normal restarts from successful updates though—no point spamming you about routine operations.

The details on setting this up are in the alerts documentation.

How It Helps

The main thing is you catch issues before your customer does. That’s worth a lot.

We had one customer who didn’t realize their deployments were in a crash loop for two days because the app was restarting fast enough that it looked “up” when they checked. With alerts, that would’ve been caught in the first 10 minutes.

Getting Started

If you’re already using Distr, alerts are available now. Vendor settings for your side, customer portal for theirs. Takes about 30 seconds to configure.

If you’re not using Distr yet and you’re dealing with software distribution to on-prem or customer-controlled infrastructure… well, we should probably talk.

Let us know what you think, we would love to hear from you or support us by leaving a star:

⭐ distr-sh/distr ⭐

Glasskube Launch week #2: The wrap up

Jake Page — Wed, 18 Sep 2024 10:15:00 +0000

And just like that, our second launch week is in the books. Trying to emulate the success of companies like Supabase and SigNoz is a tall order, especially for a team that's still new to the launch week game. But one thing's for sure: successful launch weeks are a skill acquired through experience, and finding what works specifically for your project. We learned a ton from this one, and we're confident that future launches will be even more impactful.

The best part? This launch was already a resounding success, delivering immense value. Before we dive into the key actions and outcomes, here's a quick TL;DR for those in a hurry.

TL;DR

🚀 We shared a new feature everyday on X and LinkedIn, check them out here.
💖 We got so much valuable support and feedback on Hacker News
🥉 We came in 3rd on Dev Hunt
🎉 We organised our first San Fransisco Kubernetes meetup
📈 We gathered some unexpected and exciting metrics and feedback

Let’s take each item and explore them a little bit more in detail

A feature a day 📅

The theme for this launch week was loud and clear: Glasskube is GitOps ready! Most of the features we showcased support our new GitOps functionalities in one way or another. If you remember just one thing, let it be this: thanks to our latest features, Kubernetes package management can now be handled the GitOps way.

Eager to get started? Check out the Glasskube GitOps template for a working example of what's possible.

Here's a quick recap of the features we shared:

Glasskube is "GitOps ready"
The Glasskube GitOps template
The Renovate integration
The Glasskube Hub
Glasskube package scopes

Let's dive deeper into each one!

Successful Show Hacker news launch 📰

On Tuesday, we launched the GitOps template on Show HN. We were blown away by the insightful engineers who gave it a go and came back with thought-provoking, challenging questions, the kind of feedback that's pure gold for us.

Finding smart, generous people who dedicate time to giving valuable constructive feedback is rare, and we're incredibly lucky to have found it again in the Hacker News community.

Some of this feedback even sparked GitHub issues and Discord conversations that have been incredibly fruitful and will undoubtedly leave a lasting mark on the project.

Not on the Glasskube Discord server? Join us here!

Top 3 dev hunt 🚀

Launch weeks are the perfect time to test assumptions and explore new platforms and tools. This time, we decided to give DevHunt a shot, a platform similar to Product Hunt, but tailored towards developer tools. While the audience is smaller than on sites like Product Hunt, we figured it was worth a try. After all, it's not just about quantity, but the quality of the members on the site. And I have to admit, if you're building a developer tool, you might get fewer upvotes on DevHunt, but the potential to find users who are closer to a perfect fit for your target persona is much higher. We're thrilled to have landed in 3rd place!

Our first every San Francisco meetup 🌉

Why not mix it up any throw in an IRL event into the mix too. On Thursday we celebrated our first ever Kubernetes, DevOps and Security related meetup in San Francisco with more than 100+ sign-ups.

We had three great speakers who took the stage to bring technically rich yet super egnaging talks on the following topics:

- Lukas Gentele (Loft Labs): Tenant Autonomy & Isolation In Multi-Tenant Kubernetes Clusters - A Comprehensive Guide
- Philip Miglinci (Glasskube): The State of Package Management on Kubernetes
- Ramiro Berrelleza (Okteto): Your AI team doesn't need a platform, but a paved ramp sure can help!

The turnout and the vibe were amazing. To cap the evening off we had a great networking session fueled by a generous amount of Pizza. We might have overdone it, but can you really ever have enough pizza?

If you weren’t able to miss it, don’t worry. We will be organizing more and will make sure to explore other locations also.

Thanks to everyone who joined!

Community feedback 🗣️

While there are amazing companies that pull off launch weeks with seemingly endless reach and impact, this section highlights why any project, at any stage, should embrace them. The truth is, no level of social media reach or post impressions can replace real, tangible community feedback and action.

A perfect example from this launch week is a community member who challenged the very conception of one of our newest features by spurring a conversation via GitHub discussions. This type of feedback is invaluable to us, and we genuinely welcome it. It's one of the main reasons we feel this launch week was so worthwhile.

Reaching key metrics 📊

Even though this launch week was primarily about sparking conversations with real Glasskube users and gathering feedback, we're happy to report some interesting key metrics:

Over 100% website view growth
Over 10% increase in real user installations
2 days featured on the Show Hacker News page
20+ Glasskube cloud signups
+2 new Glasskube contributors
Over 2400 impressions on Dev Hunt

Launch weeks are like a finely tuned orchestra playing in sync. Each day requires well-planned, prepared, and executed actions. As the week progresses, you build momentum, using it to amplify your message. We're excited to apply everything we've learned to our next launch. Nobody said launch weeks are easy, but with careful planning, scheduling, and execution, you can create a beautiful symphony.

Thanks for joining us on this ride, and we'll see you at the next one!

2024: A new frontier for Kubernetes package management

Jake Page — Mon, 19 Aug 2024 10:27:27 +0000

2024 has been an action-packed year for Glasskube. In February, we launched a Kubernetes manager that was initially developed internally by our team following an unsuccessful product launch that failed to take off in part due to limitations in currently used package management tooling. After this lukewarm Kubernetes operator debut, Glasskube founders Philip and Louis shared their launch post-mortem on Hacker News. They discovered that many others were experiencing similar frustrations with the existing package management solutions for Kubernetes. Recognizing this shared pain, a product pivot became inevitable, and the Glasskube package manager began to take shape.

Now, it’s only mid-August of the same year, and we are already closing in on 1.5 million downloads, 2.6k GitHub stars, and 50+ contributors. We've witnessed significant growth in adoption, features, and use cases. As we continue to evolve, we wanted to take a moment to reflect on the work we've accomplished so far. This blog post serves as a comprehensive update, a flag in the road, to share everything you need to know about the current state of Glasskube, what’s on the horizon, and how you can help shape the future of Kubernetes package management.

Welcome to the new frontier of Kubernetes Package Management.

Introduction

Why Kubernetes needs better package management

Since its inception Kubernetes as a container orchestration system has never stopped growing in popularity and its adoption keeps increasing. In recent years, the number of available packages has grown from about 100 to over 800. This growth by all means is very impressive and shows the maturity of the Kubernetes ecosystem, but it has also shined a light on the problems with current package management tools. Developers often struggle with:

Complex and highly bespoke workflows
Time-consuming and many times manual processes
An over-reliance on current package manager tooling that leave a lot to be desired.

These issues can lead to:

More work for operations teams
Non-scalable workflows
Possible security risks

It's clear that Kubernetes needs a way to manage packages that evolve in the way that modern-day Kubernetes clusters are used, that is in a more automated and overall declarative way.

What is Glasskube?

Glasskube is an open-source package manager made for Kubernetes. It aims to make installing, updating, and setting up packages easier and faster. In fact, it can be up to 20 times quicker than tools like Helm or Kustomize.

Glasskube took ideas from easy-to-use package managers like Homebrew and npm. It offers two ways to manage packages:

A graphical user interface (GUI)

A command-line interface (CLI)

glasskube serve # to access the Glasskube UI

Current tools: Helm and Kustomize

Helm and Kustomize are two main tools for managing Kubernetes packages. They help set up and run applications in Kubernetes, but work differently.

Helm uses templates to create Kubernetes yaml files. It packages apps into "charts," which are sets of YAML files. Some of it’s benefits are:

Configuring complex apps
It shines at app installation
Rolling back changes is straightforward
It’s highly adopted

Kustomize takes a different approach and many times works in conjunction with Helm. It lets users:

Set up basic app configurations
Add changes for different environments
Avoid complex templating

Problems with existing tools

While Helm and Kustomize are useful, they can cause issues for developers.

Helm can be tricky:

Hard to learn due to complex templates
Customizing apps for different setups can be confusing
CRD updates are not possible
Updates are clunky
Helm is inherently one-sided, after installation its job is done.

Kustomize has its own problems:

No built-in way to manage packages
Users need other tools to share their setups
Handling complex dependencies can be hard
Less automated than Helm for managing app lifecycles

These issues show why a simpler package manager for Kubernetes is needed. This is where Glasskube comes in, aiming to make the whole process easier.

Care to support us?

If this is the first time you've heard of Glasskube, we are working to build the next-generation Package Manager for Kubernetes.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Glasskube explained

How Glasskube works

Glasskube makes managing Kubernetes packages easier. It offers two ways to use it:

A graphical interface (GUI)
A command-line interface (CLI)

Glasskube uses a central package repository called the Glasskub Hub that:

Aware of package dependencies
Works well with GitOps
Updates packages automatically
Allows for multiple public and private package repositories to be referenced

Probably most importantly, Glasskube has client and server-side components. The server-side components constantly track the health status and desired state of the installed packages. The server-side components are the:

PackageController: communicates with the Kubernetes API to deploy and reconcile packages.
PackageInfoController: reads package manifests from the configured backend package repositories to determine the state of truth of the cluster packages.

Main features

Glasskube has several key features:

Solving common issues

Glasskube directly addressed gaps that other Kubernetes package managers have:

It's easier to use than Helm, which can be hard for new users
It handles dependencies better than Kustomize, which doesn't have built-in package management
It works well with GitOps, making it easier to:
- Roll back changes
- Upgrade apps
- Work together as a team
Package scopes help to be more efficient with package installation and package sharing. Ex. if multiple packages depend on cert-manager, one instance will be enough to serve more than one package.
Have easy access to packages with frontends without having to manually portforward

How to set up Glasskube

What you need to install

Before you start, make sure you have:

A working Kubernetes cluster, minikube is an easy alternative for a quick local setup.
kubectl installed on your computer

Step-by-step installation guide

Here's how to set up Glasskube:

Install Glasskube CLI: Open your terminal and run on MacOs:

brew install glasskube/tap/glasskube

For other installation methods click here.

Bootstrap Glasskube to install the server-side components by running:

glasskube bootstrap

Check the installation Run:

glasskube version

Make sure both Glasskube and the package-operator are installed and running the same version:

glasskube: v0.17.0
package-operator: v0.17.0

Using Glasskube: Key functions

Access the GUI

Glasskube makes it easy to add and remove packages in your Kubernetes cluster:

To add a package:

glasskube serve

Install packages

Any package available in the Glasskube hub is ready to be installed

glasskube install <package-name>

This command will then request confirmation on the package version before installing.

To remove a package:

glasskube uninstall <package-name>

Managing package dependencies

Glasskube takes care of package dependencies for you:

When you install a package, it automatically installs any required dependencies
This saves time and prevents errors from missing components

In the example below upon installing the keptn package, an instance of cert-manager will also be installed if not already present in the cluster.

Handling package configurations

You can easily change or view package settings with Glasskube, note that not all packages have custom value configurations:

In the example below, the caddy-ingress-controller has the automaticHTTPS value exposed, where you can add an email address to enable HTTPS functionality.

Working with GitOps

Glasskube works well in GitOps workflows, letting users control their Kubernetes packages desired state in Git means:

Your Kubernetes package setup stays in sync with your files stored in Git
You can use tools like ArgoCD or Flux to apply the desired package state stored in Git
Changes are easy to track and undo if needed
Cluster setup can be achieved in minutes instead of hours.

We have a fully operational Glasskube GitOps template here. Follow the steps in the README.md file to get started.

Automatic updates with Renovate

Glasskube keeps packages in Glasskube hub always updated to the latest stable versions:

Enable auto-updates or get notified when new versions are available
Shows package information including the current version and available versions for packages using glasskube list
Running glasskube update <packageName> will install the newest available version of the package
Also updating manually through the GUI is also an option.

Adding custom package repositories

You can now add multiple package repositories for Glasskube to read from, this is be useful for:

Companies that have their own private package lists
Teams who want to manage internal packages more easily
It allows following compliance rules for private packages that can’t be exposed publicly

Add a public repository by running:

glasskube repo add <name> <url>

Add private repositories

glasskube repo add <name> <url> --auth (none|basic|bearer)

Glasskube vs. other package managers

How it differs from Helm and Kustomize

Glasskube offers a new way to manage Kubernetes packages, addressing limitations found in older tools like Helm and Kustomize. It offers features such as two-way communication between the package and package manager, enhanced automation, and a more declarative approach to package management. Here's how Glasskube stands out:

Automatic Notifications: Glasskube keeps track of installed versions and can automatically notify users of new installations, if desired.
Seamless CRD Updates: It updates Custom Resource Definitions (CRDs) smoothly, ensuring everything remains in sync.
Full GitOps Integration: Glasskube enables a complete GitOps workflow, which wasn't possible before.
Efficient Control: The additional layer of abstraction through package scopes allows for more efficient and granular control over package configuration and usage.

Business service offerings

Companies that offer commercial tools with self-hosting options often leave money on the table. Custom self-hosting installations and management might not be within your company’s expertise or bandwidth. A Glasskube Native package can be the solution you need. We can collaborate with you to create standardized, customizable, and highly scalable packages tailored for a variety of self-hosted environments, all using Glasskube packaging. This approach ensures a seamless experience for your customers while maximizing your revenue potential.

If you think that Glasskube Native packages could benefit your business, book a free demo call with us. Our team would be delighted to assist you.

What's next for Glasskube

New features coming soon

Glasskube is adding new tools to make package management easier:

These new features will help users manage their Kubernetes packages better.

Wrap-up

Kubernetes package management has been stagnant and unable to keep up with the fast growth and evolution of the general Kubernetes ecosystem. Glasskube aims to make Kubernetes package management easier, more declarative, flexible and far less reliant on rigid templating processes that require multiple tools to get the job done.

As Glasskube grows, we will need help from users and developers to build in the right direction. What frustrates you the most about current Kubernetes package managers? How can we improve?

For teams using Kubernetes, we hope tools like Glasskube will be increasingly adopted and applied in complex environments.

2024 has been an incredible breakout year so far, but we have only just begun.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

The GitOps Kubernetes starter template that gets you set-up in minutes instead of hours

Jake Page — Thu, 08 Aug 2024 15:13:45 +0000

We have all heard off writers looking at a blank page for hours trying to get over writers block. For Kubernetes admins, setting up a brand new Kubernetes cluster can be just as daunting.

Especially when it involves configuring multiple elements like ArgoCD, a PostgreSQL database, monitoring tooling, and custom web apps.

However, what if there was an easier way? Our new GitOps template is designed to make this process straightforward and hassle-free. It offers an easy to set up solution that requires almost no manual steps for a basic yet highly extendable GitOps setup. With this template, you can have your cluster up and running quickly and efficiently, allowing you to focus on what matters most, building and scaling your applications.

If you’re reading this, you likely already appreciate the benefits of GitOps and have a solid understanding of Kubernetes. However, until recently, there hasn’t been an elegant, standardized process for managing the packages installed in your Kubernetes clusters. That’s why we built one.

Let’s put it to the test. With this template, ArgoCD, CloudNativePG and a simple bookmarking web app will be installed using the Glasskube Package Manager and they’ll receive upgrades through a Renovate integration via pull requests. This future-proof setup is easy to maintain and ensures your clusters stay up-to-date effortlessly.

The next logical extension for infrastructure IaC

There are clear frameworks, templates, and standards for provisioning infrastructure across almost all cloud providers, and even templates for provisioning Kubernetes clusters themselves. However, this is often where the smooth, paved Infrastructure as Code (IaC) road turns to gravel. Kubernetes admins are frequently left to their own devices to install, configure, and manage the packages within the Kubernetes cluster due to a lack of standardized and simple package management tooling.

Converting package configurations to code and managing them through GitOps, as you would with internal applications, has proven difficult. We believe the Glasskube package is a significant step toward the much-needed standardization in this area.

What is a Glasskube package?

A Glasskube package is a standardized unit for managing software deployments within Kubernetes clusters using the Glasskube Package Manager. It is defined by a PackageManifest, which contains all necessary information for identifying, configuring and installing a package. This manifest can include a Helm resource or a link to a manifest. Dependencies between packages can be declared, ensuring all required components are present before installation.

Use ArgoCD to deploy Glasskube packages

Helm charts or plain Kubernetes manifests lack the comprehensive framework that Glasskube packages offer for deploying resources declaratively in a reliable, consistent, and easily maintained way. In this demo, we’ll use the GitOps template to bootstrap a fresh Kubernetes cluster with an instance of ArgoCD, which will then be used to deploy all subsequent packages.

Here’s what the cluster will look like.

Template diagram

Template structure

🔗 Link to template

The repository contains:

a bootstrap directory containing the initial/parent Argo Application, and the necessary Glasskube manifests.
a packages directory containing the ArgoCD ,cloudnative-pg and kube-prometheus-stack cluster packages.
an apps folder containing a simple shiori bookmarking web app.
the renovate configuration file.

Glasskube custom resources will only be picked up by ArgoCD after placing the package definition files inside the packages directory.

⚠️ Please do not delete/uninstall the argo-cd package, as this will also remove everything else!

Note that the parent application used to bootstrap (bootstrap/glasskube-application.yaml) will not be synced after the initial setup. If you want to change something about it, you will have to change the application via argo directly.

Cluster setup

Prerequisites

You will have to have access to an empty Kubernetes cluster.

The easiest would be creating a new Minikube cluster with:

minikube start -p gitops

⚠️ Glasskube should not yet be bootstrapped in the cluster.

Install the Glasskube CLI

Make sure to have at least Glasskube version 0.16.0 installed locally. If you don’t, you can simply run:

brew install glasskube/tap/glasskube

How to use the template

Use this repository as your GitOps template

Create a public GitHub repository based on this starter template by clicking “Use this template“. You can move it and/or make it private afterward.

Replace the placeholder repoURL in your local GitOps repository

Replace the default value of repoURL to your repository url:

Line 12 in: bootstrap/glasskube-application.yaml
Lines 11, 16 and 26 in: bootstrap/glasskube/applicationset.yaml

Bootstrap ArgoCD and Glasskube for your Kubernetes cluster:

glasskube bootstrap git --url <your-repo>

The result

As a result, your cluster will be powered with GitOps capabilities by ArgoCD, as well as package management features by Glasskube. Argo manages itself, the Glasskube installation, as well as Glasskube packages — all of which you can now manage GitOps-style with this repo.

Run glasskube serve to open the Glasskube UI and either open the ArgoCD UI there, or with the command glasskube open argo-cd – but of course you can also use the Argo CLI. Follow the ArgoCD docs to get and reset the password to log in.

💡 Note that it might take a couple of minutes to start up ArgoCD, and for the initial GitOps sync to happen.

In this template, for demonstration purposes we’ve also installed the cloudnative-pg and kube-prometheus-stackclusterpackages and a bookmarking application (shiori), which is making use of cloudnative-pg.

Managing your cluster

Both CLI and UI offer features to manage your cluster following GitOps best practices.

Using the CLI

The relevant CLI commands offer the flags --dry-run and -o yaml which output the yaml object code which should then be pushed to your repository to be deployed.

Through the UI

The UI, when installed with the above glasskube bootstrap git command, will also output the yaml objects which you can copy to use in your git repo, instead of applying your changes directly.

Installing packages

To install a ClusterPackage, e.g. cert-manager, use the install command like this:

glasskube install cert-manager --dry-run -o yaml --yes > cert-manager.yaml

Instead of directly installing the ClusterPackage, this will write the ClusterPackage custom resource to the cert-manager.yaml file, which can now be put into a new directory packages/cert-manager/ in the git repository. Once pushed to the repo, ArgoCD will pick up the changes after at most 5 minutes, create the ArgoCD Application wrapping the Glasskube ClusterPackage. After that, the Glasskube operator will pick up the ClusterPackage and finally install it in the cluster.

Similarly, when using the Glasskube UI, one can generate the ClusterPackage resource by using the “Show YAML” button on the page of the ClusterPackage.

Updating packages

There are two options handling package version updates:

Manually

Using the glasskube update --dry-run -o yaml command, or the corresponding button on the Glasskube UI. And then pushing to your repo as seen before.

The downside of doing it that way, is that someone has to manually execute the command, even though checking for updates and preparing the updates to the git repository as an automatable task.

Automatically with Renovate

Once Renovate is integrated to track your GitOps repo, it will look for Glasskube packages and compare their versions to the official package repositories. When new versions are available, it will automatically open a PR. Once merged, you’ll be running the latest versions of your packages.

Uninstalling packages

To uninstall a package or a ClusterPackage, simply remove the custom resource from the git repository.

Updating Glasskube

When a new Glasskube version is available, the manifests have to be updated. To update the Glasskube manifests in your git repo, run the following command:

glasskube bootstrap --dry-run -o yaml --force > bootstrap/glasskube/glasskube.yaml

After reviewing and merging those changes the update will be picked up by ArgoCD. The --force flag is necessary for the command to continue manifest validation, even though failures occur.

Working with Apps

This template also contains a demo application: a bookmark manager called shiori.
Its manifests are defined in apps/shiori, which is a pattern you can follow for your own custom applications.

In a minikube environment, two manual steps are required to access the application (for more information consult the
minikube docs):

Run minikube addons enable ingress -p gitops.
Run minikube ip -p gitops and add the line <your-IP> my-shiori.example to your /etc/hosts file.

After that you will be able to access the application via http://my-shiori.example in your browser.
The default login credentials are shiori / gopher – for more information check the shiori docs.

In general, you can use the apps directory to deploy such custom applications into your cluster. Any subdirectory will be
picked up by ArgoCD and grouped as an Application.

Monitoring with `kube-prometheus-stack`

This template also installs the kube-prometheus-stack clusterpackage, which is an easy way to get started with monitoring your cluster. You can open Grafana with glasskube open kube-prometheus-stack. It does not come preconfigured in this example, but you can easily add a nice postgres dashboard and observe the metrics of the database while you are working with the bookmarking application.

Setting up a postgres dashboard

We are going to make use of the cloudnativepg dashboard.
Import it by opening the dashboard-import page, pasting
https://grafana.com/grafana/dashboards/20417-cloudnativepg/
into the first textfield, and pressing "Load". Use the "Prometheus" data source on the following screen and finish the import process.

Of course monitoring your experimental minikube cluster is a bit of an overkill, but this is simply to demonstrate how these kind of cluster administration tasks can be integrated into this gitops stack.

Template setup walkthrough

Upcoming Features

Private Repo Support

We are aware that GitOps repositories should not be public, but for simplicity we omitted this feature in the first version of the new GitOps-bootstrap command. Supporting private repos with authentication of course has high priority for the upcoming releases. We will also replace the repoURL automatically, such that you don’t need to this step manually when setting up the repo.

Improved Renovate Integration

As described above, the renovate integration currently is regex-based, and it does not consider dependencies yet. However, we don’t see these shortcomings as a blocker and recommend to try out the renovate integration in the Glasskube/Argo Gitops setup.

Improved Dependency Resolution

Installing packages with dependencies is not 100% GitOps-compatible yet, as the dependencies will be created by the operator. Consider this: To install a ClusterPackage <P> that has a dependency on D, one would do glasskube install <P> --dry-run -o yaml, which would output the ClusterPackage custom resource for <P>. However, the dependency D will only be resolved at reconciliation time by the package operator, and will therefore not be represented in the git repository at all.

A temporary workaround would be to have a closer look at the output of the install command, which also shows the dependencies which will be installed and in which version. One could then manually add the required packages custom resources to the git repo as well. However, this will be tackled in a future version to make the user experience better, see this issue.

Summary

With this template repository and guide we show how Glasskube can easily be set up in a ArgoCD powered Gitops environment, and how efficient package management is possible with this stack. Additionally we install a web application to show how custom applications can make use of the Gitops setup and Glasskube packages.

This is a first concept with some minor shortcomings, but we will continue to improve GitOps support.

Feedback

We love feedback! Whether you are just starting out or you are a seasoned professional, we'd like to hear your thoughts, inputs and questions regarding this starter template and corresponding guide here, in the glasskube/glasskube repo or on our Discord. Thanks!

Why building an IDP for Series A companies doesn't make sense

Jake Page — Wed, 07 Aug 2024 10:21:01 +0000

CTO of a Series A company? Before reading on, just give yourself a quick pat on the back. What you and your team have done so far is no easy task, so fair play to you.

Realistically though, the work is far from done. Many scaling and growth decisions lay ahead and probably more questions than clear convictions.

You might be tempted to preemptively adopt tooling and processes, positioning your engineering team for the potential growth to come. In this piece we’ll explain why you are more likely to succeed by doubling down on what brought you to this point over adopting new processes or fancy Internal Developer platforms.

Setting the right foundations

Series A companies are exciting environments to be in, usually a reasonable influx of new faces join the company, processes are increasingly standardised and overall, there is a progressive company-wide maturity that is starting to take hold. The instinct to mature the engineering org is understandable but nonetheless ill placed.

No level of maturity or sophisticated tooling can compensate for a lack of Product Market Fit (PMF). Validating PMF across markets should be top of mind at this stage above all else. Elusive as it may be, your best chance of achieving it is staying agile, delivering rapidly and maintaining tight feedback loops.

The idea of planning for future scalability might be appealing, especially with the buzz around platform engineering and the availability of ready-made internal developer platform (IDP) solutions. However, it’s crucial to understand why these options might not be the right choice for you right now. Let’s delve into the reasons why an IDP isn’t the change you think you need.

What is an IDP?

An IDP is a self-service layer build on top of your organisation’s infrastructure, that enables your developer teams to deploy and manage applications without the need to have in-depth infrastructure knowledge or relying on the help of traditional infrastructure engineers. It aims to provide consistency, automation, and speed by standardizing workflows and environments as the organization grows.

Think of an IDP as an internal product maintained by DevOps or Platform engineers, who essentially serve their own customer base, your development teams. To understand why adopting an IDP might be counterproductive for a Series A company, it’s essential to understand why IDPs emerged in the first place.

As companies expand, so do their engineering teams. In large enterprises, maintaining code quality, keeping up with infrastructure requirements and sustaining deployment velocity is hard if operations teams can’t keep up. IDPs were designed to remove reliance on traditional operations teams, to rely on a platform that actively evolves to fit their needs.

A few assumptions are important to point out here:

You have the resources to maintain a completely new product (the IDP). You probably don’t.

You have already scaled the engineering org and now need to optimise it. You probably haven’t and don’t need to.

What should Series A companies care about?

Before going on about the types of tools and processes your shouldn’t care about, let’s take a more proactive approach and focus on things that will actually move the needle.

The right team

You need a group of people who can run themselves, have the expertise to automate infrastructure related tasks and are open to working in flexible and not overly standardised structures. They should be the ones who propose changes and implement them as needed.

Finding and validating PMF

By the time you are at a Series A level you have surely caught the scent of where you might find PMF. Do not let up, ship fast and communicate intensely with your target audience to validate your market assumptions. There is no reward in finding a gap in the market alone, you have to occupy it. Maintain development velocity to increasingly occupy the gap you have identified.

Staying agile and flexible

The engineering team, like all teams within the company, should be unequivocally aligned with one primary objective: enabling the business. The engineering team sits at the centre of a short but tightly knit feedback loop which includes sales, feature ideation, feature development, deployment, and gathering feedback. Ideally, the sales team is working flat out, and customer service is maintaining close contact with customer. For the engineering team to keep it’s end of the bargain, it has to stay flexible and agile, ready to respond quickly the other teams inputs.

Focusing on customer facing products only

Even though we’ve mentioned this before, it’s important to emphasise that an IDP is a fully-fledged product that needs to be created, maintained, and evolved to meet the changing needs of internal developers over time. Focusing on an internal product like this can detract from the attention and resources needed for your customer-facing products, which ultimately bring you in closer contact with PMF and customer satisfaction.

Building the right culture

This is a point that should not be overlooked. Regardless of your company’s stage, evolution is always and ongoing dynamic. A culture of alignment, healthy communication principles, and a focus on shared goals is the bedrock that must serve as a base for the discussions about architecture, tooling, and process definitions which enable said evolution. Collective buy-in can only happen if a meaningful shared culture exists.

What should a Series A CTO not be thinking about?

It’s not un-common to hear people report having learned more from their bad managers than from their good ones. Why might that be? Because knowing what not to do can sometimes be more important the the opposite.

Don’t maintaining internal products

Any time spend not addressing requests from your product and sales teams and hopefully your customers is time wasted.

Any time spend that doesn’t involve delivering a better experience than your competitors is a mistake.

Avoid large engineering teams

Just because the rest of the company is growing doesn’t mean the engineering team has to expand at the same rate. Those who praise the magic that can be harnessed by keeping engineering teams small are on to something.

At this stage, as you may be bringing on your first DevOps engineers, ensure they integrate into a compact, developer-focused team with a single leader to avoid unnecessary managerial layers, allowing for open communication and quick decisions to be made.

Forget about perfection

It’s not news to you that shipping quickly and continuously improving over time is the best way to build a product that eventually wins market share and meets customer expectations. Why would scaling engineering teams be any different? Trying to implement an Internal Developer Platform too soon runs contrary to those principles. There will be a time when you will build one out but let that process happen iteratively in a gradual manner. You have others things to worry about in the meantime.

You are on the right track

Reaching a Series A funding round is not the end goal for anybody, you have a lot of room for growth. PMF is more than likely not locked down and occasional engineering bottlenecks should be dealt with on a case by case basis. Now is the time to double down on the strengths and keep on improving you capacity to identify your customer and increase user satisfaction.

There will be a time to look inwards and improve internal engineering processes, and IDP might make sense in the future, but at this moment in time your limited engineering output should be laser focused on staying lean.

Implementing internal products that don’t directly enable the business too soon have the potential for dire consequences. Approximately 35% of series A funded startups fail before Series B, a lot is at stake.

No need to worry though, you have made it this far, don’t give into platform engineering FOMO, put you head down and keep doing what you’ve been doing, because it’s working.

We would love to hear from you, what are your thoughts? when is an organization really ready for an IDP?

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

20 Life hacks for DevOps Engineers

Jake Page — Tue, 23 Jul 2024 09:29:37 +0000

Tricks of the trade, hacks, trade secrets, cheat sheets, best practices, call them what you will. Every industry has them, and anyone who sticks around long enough builds an arsenal of techniques and finely tuned tools to excel at their job.

Some things simply take time to master. My dad, a retired builder, could tile a medium-sized bathroom in under an astonishing three hours, while it would take me a full day just to do the grouting afterwards. Experience is key for certain skills, but there are also tips that don’t require years of practice to acquire.

DevOps is no different. There are no shortcuts to becoming a god-tier DevOps savant, put the years in and you might just get there. Having said that, there are some tricks of the trade, life hacks and useful tooling that are sure to give you an instant boost in productivity.

Here is my non-comprehensive list of life hacks guaranteed to make any DevOps engineer’s life easier.

The list is broken down into:

Tooling 🧰
Skills 🤸
Habits 🔁
Scripts, configs and extensions 💻

Tooling: 🧰

Did you know that if you really want to get somebodies attention in Germany, sending a fax is your best bet?
Also, in Japan up until this year, floppy disks were still in use by government institutions.

Knowing which context you find yourself in is essential to then choose the best tools for the job. Even though it’s important not to obsess over having the best, newest, or shiniest tool on the market. The following tools can be real game-changers for DevOps engineers.

"You don’t need to be the sharpest tool in the shed to use the sharpest tools in the shed" - Anonymous (I might have made it up)

1. k9s

K9s is a terminal-based UI for interacting with your Kubernetes clusters. It takes very little time to get accustomed to navigating, observing, and managing your live applications through the tool. And once you do, you might never go back. K9s continually monitors Kubernetes for changes and offers many useful commands to interact with your observed resources.

Link to instal.

2. tmux

tmux is a powerful terminal multiplexer that enhances productivity by allowing session persistence, window and pane management, and customization through key bindings and configuration files. It supports scripting for automation, facilitates collaboration with shared sessions, and integrates well with various shells and tools.

3. Glasskube

Glasskube is an Open Source package manager for Kubernetes. It makes deploying, updating, and configuring packages on Kubernetes 20 times faster than tools like Helm or Kustomize. Inspired by the simplicity of Homebrew and npm. You can decide if you want to use the Glasskube UI, CLI, or directly deploy packages via GitOps.

Care to support us?

If this is the first time you've heard of Glasskube, we are working to build the next generation Package Manager for Kubernetes.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

4. ripgrep

Riggrep is a powerful search tool known for its speed, flexibility, and user-friendly output. It quickly processes large codebases using advanced algorithms, supports a wide range of search patterns, and presents clear, highlighted results. Riggrep integrates well with other tools, is cross-platform, and customizable.

5. Firefox containers for multi cloud account access

Firefox Multi-Account Containers is a such an underestimated browser extension for it's utility. It helps manage online activity by separating websites into different containers or tabs, preventing sessions tracking across sites. It’s most useful feature is that it allows users to log into multiple accounts simultaneously within the same browser. By isolating sessions through cookie separation, it protects personal data and improves the overall browsing experience. Have multiple AWS accounts? Not an issues, you can log in to all of them from the same browser window.

6. VPA

Vertical Pod Autoscaler (VPA) frees users from the necessity of setting up-to-date resource limits and requests for the containers in their pods. When configured, it will set the requests automatically based on usage and there after allow proper scheduling onto nodes so that appropriate resource amounts are available for each pod.

Install it here.

Example config:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       my-app
  updatePolicy:
    updateMode: "Auto"

7. Kctx and Kubens

Is just hand’s down the most useful CLI tool for Kubernetes context and Namespace switching.

Install it here

8. ChatGPT for guidance

Use ChatGPT as if it were a senior member on your team who is never busy and always happy to answer all of your questions. Make sure your intention is deeper understanding and not blind task resolution.

Prime ChatGPT to be a senior team member with a prompt like this:

You will act as a Senior DevOps Engineer to provide life hacks and tips on how to excel in the field of DevOps. You will also be ready to help junior team members with any questions they might have. Please include practical advice, recommended tools, and best practices for managing infrastructure and continuous integration/continuous deployment (CI/CD) pipelines. Write the output using my communication style, which is clear, concise, and practical. Here are examples of my communication style:

"Focus on automating repetitive tasks to save time and reduce errors."

"Leverage tools like Docker and Kubernetes for containerization and orchestration."

"Always monitor system performance and be proactive in identifying potential issues."

"When mentoring juniors, be patient and explain concepts in simple terms."

Here are some questions you can ask ChatGPT to further fine tune the prompt:

Do you want the advice and explanations to cater more to beginner-level juniors or those with some experience in DevOps?
Could you provide more detailed examples of your communication style, especially in scenarios where you explain complex concepts to juniors?
Are there specific challenges or areas of focus within DevOps (e.g., automation, monitoring, security) that you want to prioritize for the advice and junior support?

Skills: 🤸

It's no surprise that skills can't be absorbed instantly, they require time and effort. In the ever-evolving tech world, determining which skills to cultivate can be confusing. However, as DevOps engineers, you can't go wrong with mastering scripting and prioritizing documentation.

“Build your skills not your resume.” — Sheryl Sandberg

9. Scripting

Scripting is like a swiss army knife for DevOps engineers because it automates repetitive tasks, provides essential glue between processes, and ensures consistency across environments. Learning and practicing scripting involves familiarizing yourself with tools like Makefiles, regular expressions (regex) for efficient text processing, and Bash scripting for powerful command-line operations.

Don't feel like you have to master scripting before implementing it at work. Look for manual steps in your current software or infrastructure delivery processes and try to script automations that make sense. Leverage an LLM for help if needed.

10. Documentation

Write everything down. It’s the best way to take care of your future you.

Try out a few note taking solutions here are a few:

It doesn’t matter which one you choose as long as you only choose one and you stick to it.

🤔 Don’t fall into the trap of spending too much time organising and optimising your notes. Notes don’t have to be perfect only functional. Spending too much time on note maintenance is meta-work.

Habits: 🔁

You will never get far if you rely solely on pure motivation to get things done and improve. Well-defined and consistent habits are the glue that keep you growing and effective, even when your motivation wanes. Habits seamlessly integrate tools and skills, ensuring you are effective at your job.

"Motivation is what gets you started. Habits is what keeps you going." - Jim Ryun

11. Don’t write to-do lists, block time.

I’ve been heavily influenced by Cal Newport’s writing on Deep Work and I’m quite convinced that efforts to preserve a certain amount of time per week dedicated to uninterrupted deep work is crucial for an individual who wants to contributor meaningfully to a team.

To-do list on their own are just wish lists. Once you plot them on a calendar, that’s when you have a plan.

It’s important to point out that I know very few people who execute on their time block plans with 100% fidelity every day of the week. Use them as north stars, schedule rest time when you need it. Update the list throughout the day even. But at least hold yourself to a plan.

12. Reciprocal meeting blocks

Unless you are your own boss, you’re most likely not 100% in charge of the meetings and commitments you might be obliged to participate in. Reciprocal meeting blocks is a way to counteract unexpected time commitments that just pop up in your calendar.
The idea here to reserve an equivalent deep work block every time a new meeting is added to you calendar. That way you can stay flexible and relatively available without having to compromise on your weekly deep work quota.

13. Have a shutdown routine

Especially useful if you are a remote worker. A shutdown routine is a series of questions and steps your run through to finish your working day. Ideally once you complete the checklist you should be able to forget about your work until the next day.

What I track is the following:

Did I exercise?
Did I address all of the miscellaneous tasks I had?
Do I have any open conversations?
Do I need to push any tasks to the next day?
Did I write a diary entry? (not work related but I like to get it done before I leave the desk)
How many deep hours did I do?
Did I check the metrics I’m tracking one last time before I close the laptop for the day?

14. Take notes during meetings

Taking notes during meetings and sharing them afterward should ideally be a common practice in your organization. If it isn't, this is a perfect opportunity for you to start. Not only does it ensure that no important details slip through your fingers, but it also provides a great service to your team.

15. Test run outages

Test run outages are essential for DevOps engineers to prepare for real incidents. You have to know exactly how to swiftly and efficiently connect to your clusters or VMs before an unexpected 1 a.m. alert for example, you can’t be caught off guard.

Familiarize yourself with moving files, retrieving container logs, and other critical tasks. Setting up SSH keys, kubeconfigs, and other access tools in advance will save valuable time and reduce stress during actual outages. Proactively testing and optimizing these processes ensures that you are ready to respond effectively to any disruption.

Scripts, configs and extensions: 💻

If you have to do something more than once, automate it. Why write out a full command when you can save time by using an alias? Over a long career, it might be hard to calculate exactly how much time is saved by typing "k" instead of "kubectl." One thing is for sure: it's a lot, and it's well worth it.

“You’re either the one that creates the automation or you’re getting automated.” — Tom Preston-Werner

16. Use helpful aliases

Don’t waste time typing out the full commands you write every day.

Here is a small snippet of a few of my configured aliases:

k=kubectl
kctx='kubectl ctx'
kgp='kubectl get pods'
kns='kubectl ns'
l='ls -lah'
la='ls -lAh'
ll='ls -lh'
ls='ls -G'
lsa='ls -lah'
md='mkdir -p'
rd=rmdir
run-help=man

17. Efficiently cleaning up finished jobs with TTL controller

You can specify the lifespan of finished jobs or pods before they are automatically removed by setting the .spec.ttlSecondsAfterFinished field. If you working in a job heavy environment finished jobs can quickly accumulate and become quite resource heavy.

Example config:

apiVersion: batch/v1
kind: Job
metadata:
  name: test-ttl-job
spec:
  ttlSecondsAfterFinished: 100
  ...

18. Git script to keep synced with the upstream

git remote add upstream <upstream-url>
git fetch upstream
git rebase upstream/main
git push --force-with-lease

19. Kubectl auto complete

Kubectl autocompletion lets you create an alias for kubectl. This feature saves time by reducing the need for cheat sheets and is especially useful for managing Kubernetes clusters. It's also recommended for the CKA exam due to its time efficiency.

Set up for Linux:

# install bash-completion
sudo apt-get install bash-completion

# Add the completion script to your .bashrc file
echo 'source <(kubectl completion bash)' >>~/.bashrc

# Apply changes
source ~/.bashrc

Check out other installation methods here.

20. Visual Studio Code Remote – SSH

The Remote - SSH extension lets you use any SSH-enabled remote machine for development, making it easier to develop on the same OS you deploy to, use powerful hardware, switch between environments, and debug applications remotely.

Install it here.

Final thoughts

There’s no magic formula for becoming a top 1% DevOps engineer. Like in most professions, it’s a matter of time, dedication, and experience that will gradually shape you into a highly effective professional. Over time, you'll get much better at recognizing patterns, recalling past situations, and finding quick solutions to recurring issues. So, don’t expect any single life hack from this list to instantly earn you a 50% raise and a promotion.

However, if you consistently focus on refining your tools, honing your skills, refusing to break good habits, and implementing smart automations, you’ll be well on your way to rapidly evolving and surpassing your current self. And who knows, maybe that promotion is just around the corner.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Simplify Kubernetes Monitoring: Kube-prometheus-stack Made Easy with Glasskube

Jake Page — Mon, 15 Jul 2024 10:18:11 +0000

What do we, as developers and engineers, value most above all else? The answer is simple: our time.

Tools that deliver value in the shortest amount of time have the highest chance of user adoption, it's as simple as that.

What else do most engineers value? Beautiful and data-rich dashboards.

Prometheus and Grafana are open-source, community-backed solutions with stellar reputations. They bring immense value by fetching and storing metrics while enabling the creation of dashboards that are not only useful but also easy on the eyes.

The uncomfortable truth is that anyone who has ever set up Prometheus alongside Grafana as their environment's monitoring stack from scratch has probably felt the frustration of not getting value especially quickly. Metric exporter configuration, dashboard widget customisation and deciding what to monitor and alert on in the first place takes time.

That's why Kube-Prometheus-Stack was created. It installs a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules, providing an easy-to-operate, end-to-end Kubernetes cluster monitoring solution with Prometheus using the Prometheus Operator.

This sounds like good news, and it is, but the stack is bundled in a Helm chart, and just the values.yaml file has over 4000 lines. Configuring and maintaining the Helm chart isn’t necessarily straightforward or “fun.”

With so many configuration options, we must be getting something good right? Well yeah, we are, by deploying kube-prometheus-stack we get all of this right out of the box:

Top Layer:

User: Interacts with Grafana and Kubernetes API.

Visualization and Alerting Layer:

Grafana: Connects to Prometheus for data visualization.
Alertmanager: Connected to Prometheus for alert management.
Prometheus Server: Central component collecting and storing metrics.
ServiceMonitors & PodMonitors: Define how Prometheus scrapes metrics.
Prometheus Rules: Includes both alerting and recording rules.

Exporters Layer:

Node Exporter: Collects node-level metrics.
Kube State Metrics: Collects metrics from Kubernetes API objects.
Other Exporters: Additional exporters for various applications and services.

Kubernetes Cluster:

Kubernetes Nodes: Running applications and system components.
Applications: Monitored by the Kube Prometheus Stack.

Luckily, Glasskube now supports the Kube-Prometheus-Stack. Package configuration, lifecycle management, and installation can be done in record time.

In this blog post, we will explore the steps to configure and install the Kube-Prometheus-Stack using Glasskube, wasting no unnecessary time wrestling with never-ending values files and getting you working dashboards and alerts quicker than ever before.

Requirements:

Access to a Kubernetes cluster (Minikube will be fine)
Glasskube installed
An extra screen for all the cool dashboards you are going to want to look at all the time. 🤣

Before we begin

For us at Glasskube crafting great content is as important as building great software. If this is the first time you've heard of us, we are working to build the next generation Package Manager for Kubernetes.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Create a cluster

Install Minikube then run:

minikube start

Check your installation by running:

minukube status

Desired output:

➜  ~ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Install Glasskube

If you already installed glasskube you can skip this step.
If not, glasskube can easily be installed by following your distribution's specific instructions.

MacOs:

brew install glasskube/tap/glasskube

Linux (Ubuntu/Debian)

curl -LO https://releases.dl.glasskube.dev/glasskube_v0.4.0_amd64.deb
sudo dpkg -i glasskube_v0.4.0_amd64.deb

More installation guides here

After installing Glasskube on your local machine, make sure to install the necessary components in your Kubernetes cluster by running glasskube bootstrap. For more information, check out our bootstrap guide.

Once Glasskube has been installed access the UI with:

glasskube serve

Navigate to http://localhost:8580/ to access it.

Kube-prometheus-stack installation

Installation can be done via the CLI, UI or even through a YAML package definition file. Since we will customize the deployment we will use the UI for this example.

Package Customization

Glasskube offers a series of customisations that we can be tweaked and adjusted from the CLI or GUI, saving you from having to render and configure the values.yaml file directly.

Let’s take them one by one.

Enable Alertmanager 🚨

We want Alertmanager to be enabled so we can leverage the metrics prometheus exposes to create helpful alerts.

Grafana Domain 📊

We will leave this empty for this demo since we would need to deploy an ingress controller to our cluster to handle the ingress object associated with the Grafana service. We could use Ingress-nginx or Caddy-ingress which are also supported by Glasskube for this.

Glasskube will automatically port-forward the Grafana pod so we can access the dashboard via the Open button.

Node Exporter host network 💽

Let’s also enable this to export node level metrics like memory and node level CPU usage.

Prometheus retention 📅

This is a duration in days for how long we want to persist the collected metrics.

Prometheus storage size

The amount of storage requests we consider the package will need.

Parameter input methods 🗄️

Glasskube allows for various methods of parameter input:

From a Kubernetes Secret
From ConfigMap
Value from Package Configuration
Via the UI

By choosing to inject data via Kubernetes Secrets, ConfigMaps and Package configuration we can maintain simplicity without compromising security.

Here is the example of how we would reference a specific configMap we have already created and deployed to our cluster.

💡 If you're using Kube-prometheus-stack and considering Glasskube for package lifecycle management but need support for specific key parameter customizations, please open an issue on GitHub with your use case. We'll do our best to expand the parameter list accordingly.

Install via Glasskkube

Once the configuration section is complete, install kube-prometheus-stack.

Upon installation you can see that the kube-prometheus-stack namespace has been created. And a series of pods have been deployed, including the grafana dashboard, the prometheus operator and the kube state metrics pods too.

Access the dashboards

In next weeks blog post we will access the dashboard via a custom dedicated Grafana URL

Hit the Open button or if you want to access Grafana on a different port you can simply port-forward the pod, which will map the exposed Grafana port to a port on your localhost. I've arbitrarily chosen to port-forward to localhost 52222 since it's available.

kubectl port-forward POD_NAME 52222:3000

Head over to http://localhost:52222/ and you will then be greated by the Granfana login page. To find your credentials which are stored in a Kubernetes secret that was generated as part of the deployed stack, run:

kubectl get secret kube-prometheus-stack-kube-prometheus-stack-grafana  -o go-template='
{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

Which will output something like:

admin-password: prom-operator
admin-user: admin
ldap-toml:

Upon access you we be greeted by a long list of powerful pre-configured Grafana dashboards which are already showing local cluster metrics:

Easily access CPU usage information

Here is a segment of the nifty CoreDNS dashboard that also comes preconfigured

Alerting

We already get many useful alerts created for us right out of the box.

In this snippet you can see that some of the preconfigured alerts are already firing: ↘️

If you want to be notified in via email, PagerDuty or any number of third party supported you will just need to add your contact points of preference and then add them as destination inside custom notification policies.

The Kube-prometheus-stack offers tremendous "out-of-the-box" value for Kubernetes cluster monitoring, eliminating the need to start from scratch. It bundles essential components for metrics exposure, extraction, alerting, and visualization, helping you establish a robust monitoring posture from the get-go. With official support from Glasskube, managing and updating a comprehensive, best practice-compliant monitoring stack has never been easier.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Log and trace management made easy. Quickwit Integration via Glasskube

Jake Page — Mon, 01 Jul 2024 12:14:52 +0000

Distributed application troubleshooting can be a nightmare. Unless you have the budget for expensive proprietary monitoring SaaS solutions or the expertise to run and maintain an complex ELK stack you might feel as if you are stuck in a cave without a flashlight.

Luckily, viable open-source alternatives like Quickwit are here to come to the rescue. By weaving together existing tooling for log and trace ingesting as well as pairing well with dashboard and visualisation tools such as Grafana and Jaeger. And sandwiching powerful indexing storage and search capabilities in between. Even if the tool sounds new, it won’t be for long.

We recently integrated Quickwit with Glasskube and it’s available to be easily deployed to your cluster. I spoke directly with Quickwit co-founder François Massot to get the insider scoop, and to learn how the tool works. Let's dive in!

But what is Quickwit exactly? 🤷

Quickwit is a cloud-native search engine that emerged with the goal of creating an open-source alternative to expensive monitoring software like Datadog and Splunk. Along the way, they’ve also developed and open-sourced several components, including ChitChat (cluster membership protocol), mrecordlog (WAL), Whichlang (fast language detection), witty actors (actor framework), and bitpacking (SIMD algorithms for integer compression).

Quickwit, with its robust Elasticsearch-compatible API, integrates well with tooling from the OSS ecosystem, such as Grafana, Jaeger, and OpenTelemetry. Users are successfully deploying Quickwit at scale, with hundreds of nodes and hundreds of terabytes of data ingested daily, all while enjoying significant cost reductions and how thanks to Glasskube to can get up and running in no time.

Quickwit excels in handling logs, traces, security data, and append-only datasets, with plans to support metrics soon. A key feature is the usage of object storage for the indexed data, which simplifies cluster management, cuts infrastructure costs, and enhances reliability. Multiple storage options are available such as local disk, Amazon S3, Azure Blob storage or Garage, an OSS distributed object storage, are available.

Questions for the co-founder François Massot 🙋

What are the benefits of using external Object Storage as opposed to node attached storage?

There are a lot of benefits! From the beginning, we chose to decouple compute and storage to make our search engine scalable, reliable, and very cost-efficient. If you want to remember one thing distinguishing Quickwit from traditional search engines, this is decoupled storage and computing.

Firstly, it provides elasticity, allowing us to scale storage and compute resources independently, which is ideal for cloud environments. Secondly, it’s cost-efficient, as object storage like S3 is cheaper than traditional disk storage, especially for large volumes of log data. And you don’t need to replicate your index data; this is done by the object storage layer. Additionally, it ensures high durability and availability, reducing the risk of data loss. Last, but not least, it simplifies cluster management as most of Quickiwt’s components are stateless.

Performance Comparison: Is Quickwit Faster than Elasticsearch?

It depends!

On indexing, Quickwit is generally twice as fast as Elasticsearch and uses less CPU. Our users, like Binance, report a reduction of 80% of CPU usage at indexing!

The story is different regarding querying, as Elasticsearch has all its data on a local disk, typically SSD, and Quickwit has its indexed data in very slow object storage. In this case, you can expect query time to be lower. But Quickwit's main goal is to be sub-second queries, which is perfectly fine in the observability/security domains. If we look at this indicator, Quickwit is on par with Elasticsearch and is even faster for demanding analytics queries, whereas the data is stored on object storage!

What's in store for quickwit in the future?

We have a very ambitious roadmap! Here are the key features that will be added in the following 12 months:

Distributed ingest (July 2024): High-throughput indexing on tens of thousands of indexes.
OpenSearch Dashboard support (Q3 2024): This will enable OpenSearch users to migrate seamlessly to Quickwit with their existing dashboards.
Metrics support (Q4 2024): New storage engine optimized for time series data.
Distributed SQL engine (Q1 2025): Distributed SQL engine for analytics on top of Apache Arrow, Datafusion, and Ballista.
Pipe-based query language (Q2 2025): Introduction of a flexible and powerful query language similar to SPL (Splunk Query Language)

Use cases

Log management 🪵

Quickwit is built from the ground up to efficiently index unstructured data, and search it effortlessly on cloud storage. Moreover, Quickwit supports OpenTelemetry gRPC and HTTP (protobuf only) protocols out of the box and provides a REST API ready to ingest any JSON formatted logs. This makes Quickwit a perfect fit for logs!

Distributed tracing 📊

Distributed Tracing involves monitoring application requests as they traverse various services like frontend, backend, and databases. It's instrumental for understanding application behavior and diagnosing performance issues.

Additionally, Quickwit seamlessly integrates with OpenTelemetry using gRPC and HTTP protocols (protobuf only), as well as Jaeger's gRPC API (SpanReader only). This means you can store traces in Quickwit and effortlessly query them using Jaeger's UI.

Key features 🔑

Full-text search and aggregation queries
Elasticsearch query language support
Sub-second search on cloud storage (Amazon S3, Azure Blob Storage, …)
Decoupled compute and storage, stateless indexers & searchers
Schemaless or strict schema indexing
Schemaless analytics
Grafana data source
Jaeger-native
OTEL-native for logs and traces
Kubernetes ready via Glasskube
RESTful API

Installation guide 🦮

Prerequisites

Access to a Kubernetes cluster (you can easily create a local cluster by using Minikube or Kind)
kubectl isn't strictly speaking a dependency for installing packages via glasskube, but it is the recommended way to interact with the cluster. Therefore, it is highly recommended. Installation instructions are available for macOS, Linux and Windows.

Install Glasskube

If you already installed glasskube you can skip this step.
If not, glasskube can easily be installed by following your distribution's specific instructions.

For this demo I'll be using a MacOs distribution:

brew install glasskube/tap/glasskube # install the glasskube cli
minikube start # start a minikube Kubernetes cluster
glasskube bootstrap # install glasskube on the kind cluster

For more installation guides, find them here.

Once Glasskube has been installed access via the UI with:

glasskube serve

The dashboard will open up on http://localhost:8580/.

Creating an S3-Compatible Bucket

Before installing Quickwit, you'll need to create an object storage bucket to hold your Quickwit indexes. You can use use your choice of Cloud provider such as Scaleway, AWS S3 or MinIO. Refer to our official Quickwit documentation for storage configuration details.

Here I will be creating an AWS S3 bucket to store the Quickwit indexes.

Steps:

Navigate to the AWS management console and create a new S3 bucket.
In IAM generate an API key, with S3 permissions, save the 'Access Key Id' and 'Secret Key', we will need them shortly.

Deploy Quickwit

From the Glasskube dashboard, find the Quickwit pacakge and add your custom configuration parameters.

defaultIndexRootUri: for this demo it's s3://quickwit-indexes.
metastoreUri: we won't use PostgreSQL so let's pick the same value we used for defaultIndexRootUri.
s3AccessKeyId: the "Access Key Id" from AWS we generated before.
s3Endpoint: Custom endpoint for use with S3-compatible providers. Not needed for S3 configuration.
s3Flavor: we are using the default empty value for genuine S3-compatible object storage.
s3Region: US-east-1 in my case.
s3SecretAccessKey: the "Secret Key" from AWS we generated before.

Here you can find the official Quickwit documentation for parameter completion.

It's also possible to install and configure Quickwit using the Glasskube CLI by running:

glasskube install quickwit

Once installed, you can see that a quickwit namespace has been created:

default
flux-system
glasskube-system
kube-node-lease
kube-public
kube-system
kubernetes-dashboard
quickwit

Now, check to see if the pods are running

NAME                                               READY   STATUS    RESTARTS      AGE
quickwit-quickwit-control-plane-86bd9955f7-bwm2r   1/1     Running   1 (27m ago)   29m
quickwit-quickwit-indexer-0                        1/1     Running   1 (27m ago)   29m
quickwit-quickwit-janitor-9479697ff-x4x2c          1/1     Running   1 (27m ago)   29m
quickwit-quickwit-metastore-56ff74df9f-k6d2g       1/1     Running   0             29m
quickwit-quickwit-searcher-0                       1/1     Running   1 (27m ago)   29m
quickwit-quickwit-searcher-1                       1/1     Running   0             27m
quickwit-quickwit-searcher-2                       1/1     Running   0             27m

We can try to access to the Quickwit UI using the following command:

$ kubectl -n quickwit port-forward pod/quickwit-quickwit-searcher-0 7280

Head over to http://localhost:7280. And you should be ready to go!

Create your first index

Before adding documents to Quickwit, you need to create an index configured with a YAML config file. This config file notably lets you define how to map your input documents to your index fields and whether these fields should be stored and indexed. See the index config documentation.

Let's create an index configured to receive Stackoverflow posts (questions and answers).

# First, download the stackoverflow dataset config from Quickwit repository.
curl -o stackoverflow-index-config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/stackoverflow/index-config.yaml

The index config defines three fields: title, body and creationDate. title and body are indexed and tokenized, and they are also used as default search fields, which means they will be used for search if you do not target a specific field in your query. creationDate serves as the timestamp for each record. There are no more explicit field definitions as we can use the default dynamic mode: the undeclared fields will still be indexed, by default fast fields are enabled to enable aggregation queries. and the raw tokenizer is used for text.

And here is the complete config:

# Index config file for stackoverflow dataset.
#
version: 0.7

index_id: stackoverflow

doc_mapping:
  field_mappings:
    - name: title
      type: text
      tokenizer: default
      record: position
      stored: true
    - name: body
      type: text
      tokenizer: default
      record: position
      stored: true
    - name: creationDate
      type: datetime
      fast: true
      input_formats:
        - rfc3339
      fast_precision: seconds
  timestamp_field: creationDate

search_settings:
  default_search_fields: [title, body]

indexing_settings:
  commit_timeout_secs: 30

Now we can create the index with the command:

./quickwit index create --index-config ./stackoverflow-index-config.yaml

Check that a directory ./qwdata/indexes/stackoverflow has been created, Quickwit will write index files here and a metastore.json which contains the index metadata. You're now ready to fill the index.

Continue on to the Quickwit documentation to add your first documents and execute your first search queries.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Glasskube v0.10.0 out now!

Jake Page — Mon, 24 Jun 2024 11:27:08 +0000

Welcome back to another new release blog post 🚀

This is where we cover the newest shipped features, enhancements, bug fixes and cover all of the recent Glasskube news to make sure you are fully up to speed. We have been riding a continous wave off momentum of internal feature developments as well as interest from the wider community which has led to the delivery of Glasskube v0.10.0.

Let’s check out what you can expect to find in this new minor release.

🚨 Alert: Breaking changes on the horizon ⛓️‍💥

Up until now, Glasskube packages could only be installed once per cluster, which sometimes imposed unnecessary restrictions and limited certain use cases.

From v0.10.0 onwards, the author of a package can specify a "scope," which can be either "Cluster" or "Namespaced" (with the default being "Cluster"). Based on the package scope, the Glasskube system creates either a cluster-scoped or namespace-scoped custom resource. The name of the cluster-scoped CRD is ClusterPackage, while the name for the namespace-scoped CRD is Package. This update introduces a breaking change since we previously used the Package CRD name for cluster-scoped resources. However, we decided to implement this change to align with common Kubernetes nomenclature (e.g., Role/ ClusterRole).

Additional features and UI enhancements

To assist in upgrading to v0.10.0 the glasskube purge command was added to help remove the previous installation.
The glasskube repo update command was also added to fetch the latest package manifests from configured Glasskube package repository.

Access to the full changelog here

Upgrading to v0.10.0

For first time installations, please follow the installation guide here.

To upgrade the Glasskube CLI, Install the newest binary files if you are on Linux or Windows machines.

For macOS, run:

brew upgrade glasskube

To upgrade Glasskube’s cluster components follow to upgrading guide here.

🆕 New Packages integrations available now

Quickwit

Quickwit is a cloud-native search engine that emerged with the goal of creating an open-source alternative to expensive monitoring software like Datadog/Splunk. With its robust Elasticsearch-compatible API, Quickwit integrates well with tooling from the OSS ecosystem, such as Grafana, Jaeger, and OpenTelemetry.

Users are successfully deploying Quickwit at scale, with hundreds of nodes and hundreds of terabytes of data ingested daily, all while enjoying significant cost reductions and how thanks to Glasskube to can get up and running in no time. Quickwit excels in handling logs, traces, security data, and append-only datasets, with plans to support metrics soon.

Hatchet

Hatchet is a distributed, fault-tolerant task queue which replaces traditional message brokers and pub/sub systems, built to solve problems like concurrency, fairness, and durability.

Find installation instructions here.

⏭️ Next packages to be supported

Kubeflow

The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable. Their goal is not to recreate other services, but to provide a straightforward way to deploy best-of-breed open-source systems for ML to diverse infrastructures.

Headlamp

Out of the box, Headlamp is a fully functional Kubernetes UI. By leveraging its powerful plugin system, builders can shape Headlamp to fit their bespoke use-cases, products, and environments.

Velero

Under the VMWare umbrella, Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.

📹 Updated demo video

Check out the newest Glasskube demo video delivered by Philip, where you can find the latest project updates up to v0.9.0.

If you haven’t already, head on over to the Glasskube YouTube channel where you can find the growing archive of weekly Community Calls, Release videos and even some short form content.

☁️ Join Glasskube Cloud

We are starting to build our Glasskube cloud offering to include advanced features in security, accessibility, and team collaboration. Stay updated on our progress by signing up here.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

10 Essential Books to Accelerate your Cloud Career

Jake Page — Wed, 19 Jun 2024 13:17:44 +0000

TL;DR 🤓

DevOps and cloud engineers need not have a floor-to-ceiling bookshelf full of books to enhance their skills. A few key books are all you need (at least at the beginning). Below I highlight key titles for both junior and senior DevOps and cloud engineers, focusing on improving productivity, understanding cloud native technologies and cloud infrastructure. Emphasizing practical application and continuous learning is essential for advancing a career in the rapidly evolving cloud industry.

With the vast amount of valuable and entertaining sources of Cloud Engineering information, from podcasts and YouTube channels to even movies, it can sometimes make us question the authenticity of the well-stocked bookshelves we see behind team members on our daily Zoom calls. Do they actually read those books, or are they just for show?

As someone who has always loved books for their content and entertainment value, I've found myself increasingly drawn to them in recent years for the change of mental pace they provide compared to other modern content mediums.

Unlike other technical book recommendation articles, I want to distinguish between four distinct categories of books that I find helpful to keep in mind. Not every book on this list is meant to be read from cover to cover, nor is every book a dense technical textbook that will gather dust on your shelf after a quick leaf-through.

The handful of book recommendations that have made an impact on my own cloud journey fall into the following categories:

Core concepts: 🏗️

Cloud Computing: Concepts, Technology & Architecture
The DevOps Handbook
Cloud Native Infrastructure

Reference books: 🔎

Site Reliability Engineering
AWS Fundamentals

Page turners: 📑

The Phoenix Project
Accelerate

Not cloud specific but still must reads:📍

Wiring the winning organisation
Slow productivity
Designing Data-Intensive Applications

🚨 Disclaimer: Gene Kim is going to feature prominently on this list

Before we begin

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Core concepts 🏗️

These are the types of books that provide beginners with essential core concepts and frameworks, forming a solid foundation that helps future lessons fall right into place.

1. Cloud Computing: Concepts, Technology & Architecture

Synopsis:

Originally published in 2013, this book is respected industry wide and is a comprehensive guide to cloud computing by exploring its history, models, mechanisms, architectures, and security considerations. Combining case studies with technical analysis, the book examines topics ranging from cloud-enabling technologies to cloud service level agreements.

What you will learn:

You will get a comprehensive and vendor-neutral understanding of cloud technologies, making it easier to assess various cloud solutions and providers objectively.
Fundamental cloud computing concepts, models, and mechanisms.
A deeper understanding of cloud characteristics, security threats, and risk management frameworks, crucial for designing secure and reliable cloud solutions.

Quote from the book:

"There is no greater danger to a business than approaching cloud computing adoption with ignorance. The magnitude of a failed adoption effort not only correspondingly impacts IT departments, but can actually regress a business to a point where it finds itself steps behind from where it was prior to the adoption—and, perhaps, even more steps behind competitors that have been successful at achieving their goals in the meantime."

A review that caught my eye:

Get it here.

2. The DevOps Handbook

Synopsis:

By presenting the “Three ways” as the principles for mitigating these challenges and achieving world-class performance, the book illustrates how organizations can adopt DevOps principles and practices to accelerate delivery, improve reliability, and create a more satisfying and productive work environment.

What you will learn:

The Three Ways (Flow, Feedback & Continual Learning and Experimentation).
Focus on Deployment Lead Time.
Conway’s law.

Quote from the book:

"DevOps isn't about automation, just as astronomy isn't about telescopes."

A review that caught my eye:

Get it here.

3. Cloud Native Infrastructure

Synopsis:

The authors explain key concepts like representing infrastructure through code, APIs, managing application life cycles in a cloud-native way, and ensuring security and compliance, but overall the book stresses that cloud native infrastructure is about adapting principles and processes more than specific technologies, impacting application management as much as hardware.

It guides readers on when and why to adopt these practices, outlining how they differ from traditional approaches.

What you will learn:

Cloud native infrastructure prioritizes building infrastructure with software over manual configuration.
Resiliency is paramount in cloud native infrastructure.
Cloud native infrastructure necessitates a shift in mindset and company culture.
Embracing chaos and designing for failure are essential.

Quote from the book:

"The only systems that should never fail are those that keep you alive (e.g., heart implants, and brakes)".

A review that caught my eye:

Get it here.

Reference books 🔎

Keep these books close by, preferably within arm's reach. In truth, the "Core Concepts" books could also fall into this category and vice versa. As you build your personal home library, you'll definitely want hard copies of these books. This way, you can reference them easily, make annotations, earmark important sections, and fully extract the valuable insights they offer.

4. Site Reliability Engineering (SRE)

Synopsis:

Site Reliability Engineering (SRE), a discipline pioneered by Google, offers a distinct approach to managing large-scale software systems by emphasizing the application of software engineering principles to operations. The book explores this approach in detail, providing insights into Google's production environment, SRE principles such as embracing risk and eliminating toil, and practical practices including monitoring, incident response, and capacity planning.

It emphasizes the importance of automation, simplicity in software design, and a blameless postmortem culture for continuous improvement.

What you will learn:

The core principles and practices of Site Reliability Engineering.
Gain insights into Google's production environment and the challenges of running large-scale systems.
How to apply SRE principles to their own organizations, regardless of size or technical expertise.

Quote from the book:

"If a human operator needs to touch your system during normal operations, you have a bug. The definition of normal changes as your systems grow."

A review that caught my eye:

Get it here.

5. AWS Fundamentals

Synopsis:

"AWS Fundamentals" guides readers through the essentials of Amazon Web Services (AWS), emphasizing practical application over certification preparation. It provides a comprehensive overview of core AWS services like EC2, S3, RDS, DynamoDB, Lambda, and more, categorized by function (compute, database and storage, messaging, etc.

The authors offer practical examples, use cases, configuration recommendations, and tips for each service. Additionally, the book introduces Infrastructure as Code (IaC), explaining its importance and demonstrating how to use frameworks like CloudFormation, Serverless, and CDK to provision infrastructure.

What you will learn:

Understanding the core building blocks of AWS (a lot of core cloud concepts are valid for over cloud providers).
Learning how to apply AWS knowledge in real-world scenarios.
Gaining an understanding of Infrastructure as Code (IaC).

Quote from the book:

“Learning AWS doesn’t need to be hard. It is important to focus on the basics and to understand them well. Once this is done all new services or features can be understood really well.”

A review that caught my eye:

Get it here.

Page turners 📑

Apart from teaching valuable professional lessons these books have felt as entertaining as other non technical works of fiction or non-fiction. All three left me with a strong “I need you to read this“ feeling.

6. The Phoenix project

Synopsis:

A novel (yes an actual novel) that follows Bill Palmer, a VP of IT Operations at Parts Unlimited, who is tasked with salvaging the company's failing IT project, code-named "Phoenix." As Bill navigates the challenges of a stressful work environment full of miscommunication, finger-pointing, and a looming audit, he crosses paths with Erik, a board member who introduces him to the principles of DevOps, without calling it that.

Throughout the story, Bill and his team work to implement these principles, striving to streamline their workflow and improve collaboration between Development and IT Operations. The book is special since the conceptual thrust of the book of spreading DevOps practices is delivered in the refreshing form of a fully fledged narrative novel.

What you will learn:

The three ways is also touched upon in this book (similar to the DevOps Handbook).
The importance of identifying and managing constraints to optimize the flow of work.
The importance of collaboration and communication between Development, IT Operations, and the business as a whole.

Quote from the book:

“Being able to take needless work out of the system is more important than being able to put more work into the system.”

A review that caught my eye:

Get it here.

7. Accelerate

Synopsis:

The book presents four years of research exploring the practices that contribute to high-performing technology organizations. The authors sought to identify the capabilities that drive software delivery performance and, in turn, impact organizational performance. Their findings highlight twenty-four key capabilities, categorized as continuous delivery, architecture, product and process, lean management and monitoring, and cultural, that demonstrably improve software delivery and overall business outcomes.

The book emphasizes that these capabilities are measurable and improvable, offering guidance for organizations to assess their current state and embark on a journey of continuous improvement.

What you will learn:

Software delivery performance is a key predictor of organizational performance
A set of capabilities, including technical practices, Lean management, and a generative culture, drive improvements in software delivery performance.
Transformational leadership is crucial in enabling and amplifying the adoption of these capabilities.

Quote from the book:

"The most important characteristic of high-performing teams is that they are never satisfied: they always strive to get better."

A review that caught my eye:

Get it here.

Not cloud specific but still must reads📍

Not specifically cloud related at all but these are book that for different reasons I feel can round out a cloud engineer.

8. Wiring the Winning Organization

I'm a huge fan of reading "leadership" or "management" focused books, even though I'm not in either of those positions. Learning what leadership should care about and the key ingredients needed to achieve excellent results serves as a cheat sheet for any individual contributor wanting to stand out by being impactful. Change doesn’t always have to come from the top, you can push it from wherever you find yourself.

Synopsis:

The book presents a new theory of performance management, emphasizing how leaders can create the conditions for their organizations to achieve exceptional results. The book introduces three key mechanisms for building a "winning organization": slowification (making problem-solving easier), simplification (making problems easier to solve), and amplification (making problems more visible).

Through a combination of theoretical explanations, practical case studies, and real-world examples, the authors demonstrate how these mechanisms can be applied across diverse industries and organizational contexts to achieve superior performance.

What you will learn:

Greatness in any endeavor is achievable through a focus on refining the "social circuitry" of an organization.
Three key mechanisms slowification, simplification, and amplification can be employed to move an organization from a "danger zone" to a "winning zone".
Leaders who embrace these mechanisms can create organizations that achieve extraordinary results.

Quote from the book:

“Slowification enables the shift from reactive, "fast thinking" based on ingrained habits to more effective "slow thinking" that allows for deliberation, reflection, and creativity in problem-solving.“

A review that caught my eye:

Get it here.

9. Slow productivity

Synopsis:

Slow Productivity by Cal Newport challenges the modern obsession with visible busyness as a measure of productivity, what he terms pseudo-productivity. Instead, the book champions a slow productivity philosophy based on three core principles: doing fewer things, working at a natural pace, and obsessing over quality.
This philosophy argues that by intentionally limiting workloads, embracing a sustainable work pace, and prioritizing quality over quantity, knowledge workers can achieve greater meaning and produce superior results.

The book explores the theoretical underpinnings of these principles and offers practical strategies for implementing them in various professional settings.

What you will learn:

To do less
Work at the more natural pace
Obsess over quality

Quote from the book:

"for all of our complaining about the term, knowledge workers have no agreed-upon definition of what “productivity” even means."

A review that caught my eye:

Get it here.

10. Designing Data-Intensive Applications

As cloud engineers, we collaborate closely with the developers in our organization who write the applications we deploy and maintain. While this book doesn’t provide directly actionable knowledge for our day-to-day tasks, it has given me a deeper understanding of the concepts underpinning the creation of our organization’s apps. More importantly, it has provided me with a common language to effectively communicate with the developers on my team.

Synopsis:

Designing Data-Intensive Applications, by Martin Kleppmann, explores the fundamental principles and practical considerations for building reliable, scalable, and maintainable data systems. The book examines various data models, including relational, document, and graph-based models, and their respective query languages, analyzing their suitability for different applications.

It also examines storage engines, data encoding formats, and schema evolution. A key focus is the exploration of distributed data systems, including replication, partitioning (sharding), and the challenges of maintaining consistency and consensus in such environments. The book uses real-world examples of successful data systems to illustrate key concepts and trade-offs.

What you will learn:

Various data models, including relational, document, and graph-based models, and their respective query languages, analyzing their suitability for different applications.
Storage engines and how databases arrange data on disk to find it again efficiently
Reliability, scalability, and maintainability.

Quote from the book:

"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair."

A review that caught my eye:

Get it here.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Glasskube reaches 1k stars 🌟

Jake Page — Mon, 10 Jun 2024 09:04:10 +0000

I was trying to thinking of sayings or mantra’s ambitious people repeat to themselves to keep themselves motivated. A few came to mind:

"Fortune favors the bold."
"No guts, no glory."
"You miss 100% of the shots you don't take."
"Nothing ventured, nothing gained."

Many of us have heard these sayings or something similar before, what you have less often which I think is equally important maybe doesn’t fit that well into a motivating one liner but might go something like this:

Sure, shoot for the stars but make sure to celebrate every win along the way. Because the journey is equally as important as the eventual goal.

While nobody starts a project with the sole aim of getting 1,000 GitHub stars, the entire Glasskube team is incredibly proud of reaching this milestone so quickly. I wanted to take a moment to reflect on our rapid organic growth and celebrate our achievements.

What is Glasskube?

Glasskube is the next-generation Kubernetes package manager, now available in its beta version. With Glasskube, you can effortlessly install, upgrade, configure, and manage your Kubernetes cluster packages. Check out the available packages here. Glasskube streamlines repetitive and cumbersome maintenance tasks, making cluster management a breeze. The project is evolving rapidly, with new functionalities being shipped with every weekly release.

So go ahead, dive in, explore, and share your thoughts with us! Your feedback is incredibly valuable as we strive to make Glasskube the best Kubernetes package manager out there. We're also beginning to develop Glasskube Cloud, building on the requests from our current users to enhance the tool further. We're eager to hear which features you'd like to see included in the Glasskube Cloud offering.

The best way to stay in the loop is to sign up for Glasskube Cloud and be informed when it's ready: https://glasskube.cloud/.

The story of our organic growth so far

Having oficially launched the project in February 2024 we are super happy to have hit the 1000k milestone in the first days of June.

Let’s take a look at some of the work be have been doing to make this happen.

We’ve been shipping 🚢

In the past few months, our team and a growing legion of open-source contributors have helped us ship eight minor version releases, the latest being v0.8.0. You can find the full changelog here. Jakob and Christophe, both Glasskube developers, have been hard at work rolling out new features and functionalities. They've also been invaluable in providing feedback and managing the PR review process, ensuring that code is merged swiftly and safely.

Every contributor, found on the Glasskube Discord server, has brought something valuable to the community. However, a few deserve a special shoutout: Hanshal, Utkarsh, and Baalakshan. These contributors have been diligently submitting valuable PRs, addressing open issues, and advocating for Glasskube wherever they go.

We've been organizing weekly community calls every Monday on the Discord server. These calls have been a fantastic opportunity to share news, align expectations, and grow together. Some contributors have even given short talks exploring concepts like Kubernetes CRDs, presented by Hanshal, and Chaos Engineering, shared by Baalakshan.

Meetups

Cofounders Philip and Louis, myself and Community members have been really active attending meetups and conferences around Europe and India. We were at a few KCD’s, AWS Summit Madrid and other CNCF meetups too. It has been a great way to spark conversations and share the work that’s being done.

Partnerships 🫂

We have partnered with third party packages such as Keptn and Quickwit to expedite their integration onto Glasskube.

You can now find installation steps right in the documentation of the supported packages.

Content creation 📚

As we build and continually refine a quality product to support our ever-expanding user base, we aim to be a consistent source of valuable content. This includes in-depth Kubernetes topics like this, more technical product explorations like this, and even broader opinion pieces like our Git guide and our article on the current state of the DevOps scene.

Check out our latest launch video 🚀

Building Glasskube so far has been such a gratifiying experience, on the one hand we are connecting and understanding the issues so many cloud practitioners are having in there efforts to deal with Kubernetes Package management in their daily routines. Better understanding them and delivering on their requests is special.

On the other hand, collaborating with so many stellar open-source community contributors is a massive perk and added bonus of building an OSS tool in public. Much of our early growth can be directly attributed to these outstanding OSS contributors. We hope to continue this collaboration as we aim for 2, 3, and 10k stars, and more importantly, to make Glasskube a tool installed in Kubernetes clusters far and wide.

If you get value from the work we do, we'd appreciate it if you could
⭐️ Star Glasskube on GitHub 🙏

DEV Community: Distr

Should I Run Plain Docker Compose in Production in 2026?

Where Plain Docker Compose Fits in Production

Docker Compose Orphan Containers and --remove-orphans

Pruning Docker Images and Capping Container Logs

Docker Health Checks Don't Restart Unhealthy Containers

Pinning Docker Images by Digest Instead of :latest

Why Mounting /var/run/docker.sock Is a Security Risk

Updating Docker Compose Deployments Across Customer Hosts

Outgrowing Docker Compose: Kubernetes vs Swarm

So, should you run plain Docker Compose in production?

Alerts for self-hosted customer deployments

The Visibility Gap

What We Built

How It Helps

Getting Started

Glasskube Launch week #2: The wrap up

TL;DR

A feature a day 📅

Successful Show Hacker news launch 📰

Top 3 dev hunt 🚀

Our first every San Francisco meetup 🌉

Community feedback 🗣️

Reaching key metrics 📊

2024: A new frontier for Kubernetes package management

Introduction

Why Kubernetes needs better package management

What is Glasskube?

Current tools: Helm and Kustomize

Problems with existing tools

Care to support us?

Glasskube explained

How Glasskube works

Main features

Solving common issues

How to set up Glasskube

What you need to install

Step-by-step installation guide

Using Glasskube: Key functions

Access the GUI

Install packages

Managing package dependencies

Handling package configurations

Working with GitOps

Automatic updates with Renovate

Adding custom package repositories

Glasskube vs. other package managers

How it differs from Helm and Kustomize

Business service offerings

What's next for Glasskube

New features coming soon

Wrap-up

The GitOps Kubernetes starter template that gets you set-up in minutes instead of hours

The next logical extension for infrastructure IaC

What is a Glasskube package?

Use ArgoCD to deploy Glasskube packages

Template structure

Cluster setup

Prerequisites

Install the Glasskube CLI

How to use the template

The result

Managing your cluster

Using the CLI

Through the UI

Installing packages

Updating packages

Manually

Automatically with Renovate

Uninstalling packages

Updating Glasskube

Working with Apps

Monitoring with kube-prometheus-stack

Setting up a postgres dashboard

Template setup walkthrough

Upcoming Features

Private Repo Support

Improved Renovate Integration

Improved Dependency Resolution

Summary

Docker Compose Orphan Containers and `--remove-orphans`

Pinning Docker Images by Digest Instead of `:latest`

Why Mounting `/var/run/docker.sock` Is a Security Risk

Monitoring with `kube-prometheus-stack`