DEV Community: Jake Page

Four Layers of Validation in Kubernetes with Claude Code

Jake Page — Thu, 21 May 2026 13:37:08 +0000

Earlier this year, Moltbook, a social network for AI agents, launched, trended, and became a cautionary tale within the same week. Security researchers at Wiz found a Supabase API key sitting in its client-side JavaScript, which was the database’s only access control, with no Row Level Security to narrow what that key could reach. The result: 1.5 million API tokens, 35,000 email addresses, and thousands of private messages exposed to anyone with a browser console.

Moltbook was a greenfield project with no review process. The same class of mistake is far more serious when AI-generated code lands inside applications that already have pre-existing users, real data, and an existing trust surface. That’s increasingly the reality: as organizations adopt AI coding agents, more and more AI-generated code is landing directly in production services that already hold credentials and personal details of your users. A recent survey found that 95% of developers don’t fully trust AI-generated code, while only 48% consistently review it before committing, yet it’s shipping regardless.

Static review tools catch only some classes of issues: common CVEs, dependency hygiene, style violations, deterministic anti-patterns. What they can’t see are things like the actual name of your Kubernetes Secret in this cluster, whether your auth middleware is wired into the right route in this service, whether the request a real user will send makes it through the new code path without breaking something downstream.

Closing that gap takes four independent layers: AI agent skills that shape what gets generated, commands that audit what was generated, integration tests that hit staging endpoints, routing traffic to your local code, and preview environments that let a human review the change against staging dependencies before merging.

Layer 1: Skills (passive, shaping what gets generated)

Most AI coding assistants let you write down rules that shape generation. The simplest mechanism is a config file that the assistant loads into context on every prompt: .cursorrules in Cursor, CLAUDE.md in Claude Code, .github/copilot-instructions.md in GitHub Copilot. Drop NEVER/ALWAYS rules in there and the AI follows them. The downside is that those files load on every prompt, even when you’re working on something unrelated, and every rule you add costs tokens whether or not it’s relevant.

Claude Code goes a step further with skills: structured rule sets that ship as plugins (a directory with a SKILL.md and supporting reference files). Each skill has a description, and the model pulls a skill into context only when your prompt matches what that skill is meant to cover. If a skill never gets matched against the prompt, it never gets loaded, and you don’t pay for the tokens it would consume. We’ve already shipped six skills for AI agents working with mirrord, and this post adds a seventh focused on validation: k8s-validation, an open-source set of NEVER/ALWAYS rules for code that runs inside a Kubernetes cluster.

The skill covers two halves: the cluster-level concerns (Secrets, RBAC, pod hardening, NetworkPolicies, supply chain, file handling) and the application-level concerns that determine whether the workload behaves correctly inside the cluster (HTTP and parameter handling, auth, output sanitisation, API contracts, env-var configuration, test coverage).

Before and after

Without the skill, the model has no way of knowing about things like Kubernetes Secrets already mounted as environment variables in your pod, auth decorators other handlers in your service already use, or PII-sanitization utilities your team has already built. So it does the obvious thing: hardcodes the API key, skips the auth check, and returns the LLM output directly. With the skill loaded, the following prompt (“add a /summarise endpoint that calls OpenAI’s API”) produces something like this:

import os
from openai import OpenAI
from utils.sanitize import filter_pii

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

@app.route('/summarise', methods=['POST'])
@require_auth
def summarise():
    text = request.json.get('text', '')
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": f"Summarise: {text}"}],
    )
    summary = response.choices[0].message.content
    return jsonify({"summary": filter_pii(summary)})

filter_pii here is a stand-in for any utility that strips personally identifiable information (names, SSNs, emails, etc.) out of free text. Different teams build this differently; the Microsoft Presidio library is one of the more common open-source starting points.

Three differences from what the model would produce without the skill: the key comes from an environment variable backed by a Kubernetes Secret, the endpoint sits behind @require_auth, and the LLM output runs through filter_pii before going back to the user.

What skills can’t do

Skills shape generation, but they don’t verify anything.

In the example above, the AI correctly followed the skill: it read the API key from the environment, applied @require_auth, and called filter_pii from utils.sanitize. But the skill has no way to verify that filter_pii actually works. If the utility in your codebase only strips email addresses and misses phone numbers, the skill can’t know that. A user document containing a phone number sails straight through the filter and into the response, and the code looks correct at every layer the skill can see.

Skills set a floor by preventing the obvious structural mistakes. They’re instructions to a model, not checks against reality.

Layer 2: Commands (active, checking what was generated)

Where skills shape what the model generates, commands check what already exists. They’re explicitly invoked by a developer, an agent, or a CI step, and they run a defined set of checks against the code in front of them.

The same install from Layer 1 also ships a slash command: /k8s-validation:audit. It scans your codebase for the same NEVER/ALWAYS rules the skill enforces during generation, traces data flow through handlers and queries, and classifies each finding by severity. Skills don’t always load (a vague prompt, or a quick edit to a file the model didn’t classify as Kubernetes-adjacent, and the rules never enter context). The audit is the backstop: it runs on the code regardless of how the code got written.

> /k8s-validation:audit content-api/

CRITICAL 2 | HIGH 4 | MEDIUM 1 | INFO 0

[CRITICAL] src/routes/summarise.py: Hardcoded OpenAI API key → Use os.environ
[CRITICAL] src/routes/download.py: User filename in path without sanitization → Use secure_filename()
[HIGH]     src/routes/summarise.py: No authentication middleware → Add @require_auth
[HIGH]     k8s/deployment.yaml: No SecurityContext defined → Add runAsNonRoot, drop ALL
[HIGH]     src/routes/summarise.py: Reads OPENAI_API_KEY but no manifest defines it → Add to deployment.yaml env block
[HIGH]     src/routes/summarise.py: New endpoint with no integration test → Add test in tests/integration/

Note that the output mixes security findings (hardcoded key, missing SecurityContext) with correctness findings, meaning “does the code do what was asked, given the rest of the system” (the Kubernetes deployment manifest doesn’t define the env var the code reads; the new endpoint shipped without an integration test). Both halves matter for AI-generated code.

Because the audit is a command you run rather than a rule the model loads, the same invocation works in three places: a developer runs it before opening a PR, an agent runs it as part of its own loop after generating code, and CI runs it as a merge gate. You can wire it into one, two, or all three.

## Validation Workflow
After generating or modifying any Kubernetes-related code, run `/k8s-validation:audit`
on the changed files. If any CRITICAL findings exist, fix them before proceeding.

What commands can’t do

The audit is still static analysis. It can find “you hardcoded a secret” or “you’re missing a SecurityContext,” but it can’t tell you whether your filter_pii regex actually catches the PII your users will send, or whether the environment variable you’re reading will resolve to a value in your staging cluster. Commands check the shape of the code, not the behavior.

Layer 3: Integration tests (runtime, proving it works)

Your team probably already has integration tests that hit your API endpoints, check response shapes, and verify that authentication rejects bad credentials. These tests encode what “correct behavior” actually means for your application.

The bottleneck is running them. Locally, you mock your database, your auth service, your message queue, and hope the mocks match reality. In CI, each cycle takes 5 to 10 minutes. For a human pushing a few times a day, it’s already frustrating enough. For an AI agent trying to fix a failing test, it’s a feedback loop far too slow to learn from: the agent burns tokens on every iteration, and the integration bugs only surface after the change has been written, pushed, and built.

mirrord changes this equation by letting your local process stand in for a deployed pod. Your local code gets the environment variables from the target pod, the same cluster-level files it has access to, and the same view of internal services. In steal mode, traffic destined for the targeted pod is intercepted and routed to your local process instead of whatever’s deployed in staging. Your existing integration tests, pointed at staging endpoints as usual, now run against your local code in seconds, not minutes.

The same pattern scales horizontally. Because mirrord can split a single pod’s incoming traffic between many local processes using header-based filters, multiple agents (or developers) can iterate against the same staging cluster simultaneously, each one routing its own slice of the traffic to its own local code. One staging environment, many concurrent agents, real downstream services for all of them.

What this catches that the other layers can’t

Consider a prompt like “have /summarise fetch the document from our content-store service first.” The agent writes a handler that calls http://content-store/documents/{id} and reads response.json()["title"].

The catch: content-store moved to v2 months ago and now returns {"document": {"name": ..., "text": ...}}. The flat title/body shape only exists in the AI’s training data. Skills generated structurally clean code (good). The audit confirmed the call was made and the response was consumed (also good). Neither layer knows what shape content-store actually returns today.

The setup to fix this is two processes. You or your AI agent starts a mirrord session, your e2e tests run as normal against the staging content-api endpoint:

# Terminal 1, run your local content-api in place of the deployed pod
mirrord exec --target deploy/content-api --steal -- python -m content_api

# Terminal 2, run the existing integration suite against staging as usual
pytest tests/integration/test_summarise.py

When the test hits staging’s content-api endpoint, mirrord steals the request and reroutes it to your local process. The local handler calls http://content-store/documents/..., and that outbound call also routes through mirrord, hitting the real content-store in staging. The real service returns {"document": {"name": ..., "text": ...}}. The local code does response.json()["title"] and crashes with KeyError.

You fix the code to read the new shape, rerun the test, it passes. The bug surfaces in your local code, against real downstream services, in seconds, instead of after a deploy cycle. The same pattern works for any other dependency the code touches: environment variables from the pod, files from mounted volumes, database queries against the real Postgres. mirrord runs your code, the cluster supplies its real environment.

Layer 4: Human review in a real environment

When the agent opens the PR, a human should still get to see the change running in a real environment, not just read the diff. mirrord’s Preview environments make that easy: a GitHub Action spins up an isolated pod in your staging cluster running the PR’s code, connected to all the real downstream services, and scoped to that PR via an environment key.

Reviewers can click through the actual feature instead of inferring behavior from the code or having to run the whole application locally. Most of the failures the previous three layers don’t catch, UX regressions, surprising interactions, “looks right but feels wrong”, show up the first time a human uses the thing.

Putting it together: agents handle layers 1–3, humans handle layer 4

Layers 1 through 3 can be run by the agent itself. Instead of generating code, opening a PR, and hoping CI catches the issues, the agent generates code shaped by skills, runs /k8s-validation:audit to check for structural issues, runs the integration tests via mirrord against real infrastructure, and fixes any failures before committing. The agent doesn’t even have to write the test from scratch: Layer 1’s validation skill includes a rule that any new HTTP handler must come with an integration test, so the test gets generated alongside the endpoint. Layer 3 just runs it against real infrastructure.

Layer 4 is the handoff. Once the agent has passed its own checks, it opens a PR and a human gets a live preview environment to click through rather than a diff to infer from. The failures that surface at that stage, UX regressions, surprising interactions, things that look right but feel wrong are exactly the ones that didn’t show up in the previous three layers.

We’ve documented the concrete setup (per-service mirrord configs, helper scripts, and an AGENTS.md that tells the agent which script to run) in our mirrord with AI agents guide. The broader argument for why this matters, including the token cost of agents stuck in a feedback-less loop, is in How to prevent token burn using mirrord with e2e tests.

Adopting the layers

The four layers are independent. Pick the ones that close your biggest gap and add the others when the next failure teaches you what’s still missing.

Install the validation skill with /plugin install k8s-validation@metalbear-co/k8s-validation-plugin in Claude Code, or as a local rules directory for Cursor and GitHub Copilot (see the repo README). The /k8s-validation:audit command ships in the same install. For the runtime layer, install mirrord and wrap your local service with mirrord exec --target deploy/<service> --steal -- <run-command>. For preview environments on each PR, the feature is included in mirrord for Teams.

Each layer closes a different gap. Stop at any point and you’ve made things better than they were.

The skills are open source. If your AI assistant generates something the skills don’t catch, open a PR.

Skin In The Game

Jake Page — Sat, 10 Jan 2026 19:29:56 +0000

A note to self to stop explaining technology and start building with it.

I’m a pretty lucky guy. Curiosity has put me in incredible situations throughout my life. I’ve learned languages, lived in interesting places, and about six years ago I decided I wanted to learn how the internet worked. That curiosity grew into a career in DevOps, cloud, and Kubernetes that I’m really proud of.

Today, as a DevRel, curiosity drives everything I do. It’s a role that changes a lot from company to company, and in my experience it even changes week to week. One week I’m preparing a new use case for a blog post or conference talk, the next I’m collaborating with colleagues on webinars and content initiatives, or acting as a bridge between marketing, product, and customer success. It’s a job where curious people thrive, and that’s been my experience too.

There’s a downside though, or at least there has been one for me. My career didn’t include a long, distinct “developer phase.” I didn’t spend years building products from scratch, shipping them, breaking them, and living with the consequences. That’s not a prerequisite for a meaningful career in tech, but it is a gap I’ve become increasingly aware of.

As my curiosity about infrastructure, Kubernetes, and cloud native tooling deepened, so did my awareness of the limits of my perspective. I could explain systems, patterns, and trade offs, but I wasn’t always the one feeling them. I was close to the work, but not always inside it.

That gap becomes most obvious when I need an example of a business case during a talk for example. When I explore a new Kubernetes architecture or a genuinely interesting developer tool, I find myself falling back on the same safe demos, a todo list, a polling app, a habit tracker. They’re familiar, low risk, and easy to explain, but they’re also uninteresting and far too theoretical.

I work for MetalBear, the company behind mirrord, a tool that fundamentally changes how developers work with Kubernetes. It lets your local process interact with real systems, real traffic, real dependencies in a remote kubernetes environment, without having to containerize or trigger CI/CD. And yet, too often, I demonstrate it using those same tired stock applications, adding dark mode to a simple polling app doesn’t do justice to tools designed for complex, high stakes environments.

At times, it’s made me feel less like a practitioner and more like a historical tour guide, passively explaining how things work, how others use them, what could be built, without actively shaping what exists today.

And honestly, who wants to be a historian of something that happened last week when you can help influence what happens next?

What I want now is simple, I want to be a DevRel with a bias for action.

If for no other reason than timing, this feels like the moment to do it. With the sheer volume of information, resources, and AI tools available today, the distance between an idea and a working project has never been smaller. Feedback loops are tighter, tooling is better, and the cost of trying and failing is lower than it’s ever been.

I want to stop talking about Kubernetes and cloud native tooling purely as a commentator and start engaging with them as someone who has skin in the game. Not just as a developer, but as someone responsible for identifying a real problem, articulating a solution, choosing the right tools, and seeing the entire process through from idea to delivery. I want to build projects that don’t just stand up in a conference talk, but stand up to real users, deal with real constraints, and real trade offs.

Some of those projects will fail. Some will stall. Some might accidentally work. That’s fine. The point isn’t success, it’s participation, it’s having actual skin in the game. It’s using the tools I talk about every day in pursuit of something useful to someone that actually could exist.

It won’t be easy to fully inhabit the skin of a developer or a product owner, and that’s exactly the point. I won’t just shoehorn a given tool or tech stack into a project because a conference calls for it. I’ll come at it from the maker perspective and determine the best way forward for any given project.

I don’t particularly care what label this puts on me. Developer. DevRel. DevOps engineer. None of that matters much. What matters is choosing to act, to build things that matter to someone, to put ideas into the world before they’re fully polished, and to learn by being exposed to reality rather than protected from it.

Building in public feels like the most honest way to do that. It raises the stakes, improve my capacity for real empathy, and it’ll force clarity. If I want to understand developers better, the best way forward isn’t more explanations, it’s shared experience.

I plan to share this journey openly and regularly, what I’m building, what works, what doesn’t, and what I learn along the way. If this resonates with you, I’d genuinely love to hear from you. If you’re also feeling the pull toward doing more and talking less, tell me what you’re thinking about building. What are you trying to validate, and what kind of value are you hoping to put into the world?

Happy 2026 everyone!

Glasskube Launch week #2: The wrap up

Jake Page — Wed, 18 Sep 2024 10:15:00 +0000

And just like that, our second launch week is in the books. Trying to emulate the success of companies like Supabase and SigNoz is a tall order, especially for a team that's still new to the launch week game. But one thing's for sure: successful launch weeks are a skill acquired through experience, and finding what works specifically for your project. We learned a ton from this one, and we're confident that future launches will be even more impactful.

The best part? This launch was already a resounding success, delivering immense value. Before we dive into the key actions and outcomes, here's a quick TL;DR for those in a hurry.

TL;DR

🚀 We shared a new feature everyday on X and LinkedIn, check them out here.
💖 We got so much valuable support and feedback on Hacker News
🥉 We came in 3rd on Dev Hunt
🎉 We organised our first San Fransisco Kubernetes meetup
📈 We gathered some unexpected and exciting metrics and feedback

Let’s take each item and explore them a little bit more in detail

A feature a day 📅

The theme for this launch week was loud and clear: Glasskube is GitOps ready! Most of the features we showcased support our new GitOps functionalities in one way or another. If you remember just one thing, let it be this: thanks to our latest features, Kubernetes package management can now be handled the GitOps way.

Eager to get started? Check out the Glasskube GitOps template for a working example of what's possible.

Here's a quick recap of the features we shared:

Glasskube is "GitOps ready"
The Glasskube GitOps template
The Renovate integration
The Glasskube Hub
Glasskube package scopes

Let's dive deeper into each one!

Successful Show Hacker news launch 📰

On Tuesday, we launched the GitOps template on Show HN. We were blown away by the insightful engineers who gave it a go and came back with thought-provoking, challenging questions, the kind of feedback that's pure gold for us.

Finding smart, generous people who dedicate time to giving valuable constructive feedback is rare, and we're incredibly lucky to have found it again in the Hacker News community.

Some of this feedback even sparked GitHub issues and Discord conversations that have been incredibly fruitful and will undoubtedly leave a lasting mark on the project.

Not on the Glasskube Discord server? Join us here!

Top 3 dev hunt 🚀

Launch weeks are the perfect time to test assumptions and explore new platforms and tools. This time, we decided to give DevHunt a shot, a platform similar to Product Hunt, but tailored towards developer tools. While the audience is smaller than on sites like Product Hunt, we figured it was worth a try. After all, it's not just about quantity, but the quality of the members on the site. And I have to admit, if you're building a developer tool, you might get fewer upvotes on DevHunt, but the potential to find users who are closer to a perfect fit for your target persona is much higher. We're thrilled to have landed in 3rd place!

Our first every San Francisco meetup 🌉

Why not mix it up any throw in an IRL event into the mix too. On Thursday we celebrated our first ever Kubernetes, DevOps and Security related meetup in San Francisco with more than 100+ sign-ups.

We had three great speakers who took the stage to bring technically rich yet super egnaging talks on the following topics:

- Lukas Gentele (Loft Labs): Tenant Autonomy & Isolation In Multi-Tenant Kubernetes Clusters - A Comprehensive Guide
- Philip Miglinci (Glasskube): The State of Package Management on Kubernetes
- Ramiro Berrelleza (Okteto): Your AI team doesn't need a platform, but a paved ramp sure can help!

The turnout and the vibe were amazing. To cap the evening off we had a great networking session fueled by a generous amount of Pizza. We might have overdone it, but can you really ever have enough pizza?

If you weren’t able to miss it, don’t worry. We will be organizing more and will make sure to explore other locations also.

Thanks to everyone who joined!

Community feedback 🗣️

While there are amazing companies that pull off launch weeks with seemingly endless reach and impact, this section highlights why any project, at any stage, should embrace them. The truth is, no level of social media reach or post impressions can replace real, tangible community feedback and action.

A perfect example from this launch week is a community member who challenged the very conception of one of our newest features by spurring a conversation via GitHub discussions. This type of feedback is invaluable to us, and we genuinely welcome it. It's one of the main reasons we feel this launch week was so worthwhile.

Reaching key metrics 📊

Even though this launch week was primarily about sparking conversations with real Glasskube users and gathering feedback, we're happy to report some interesting key metrics:

Over 100% website view growth
Over 10% increase in real user installations
2 days featured on the Show Hacker News page
20+ Glasskube cloud signups
+2 new Glasskube contributors
Over 2400 impressions on Dev Hunt

Launch weeks are like a finely tuned orchestra playing in sync. Each day requires well-planned, prepared, and executed actions. As the week progresses, you build momentum, using it to amplify your message. We're excited to apply everything we've learned to our next launch. Nobody said launch weeks are easy, but with careful planning, scheduling, and execution, you can create a beautiful symphony.

Thanks for joining us on this ride, and we'll see you at the next one!

2024: A new frontier for Kubernetes package management

Jake Page — Mon, 19 Aug 2024 10:27:27 +0000

2024 has been an action-packed year for Glasskube. In February, we launched a Kubernetes manager that was initially developed internally by our team following an unsuccessful product launch that failed to take off in part due to limitations in currently used package management tooling. After this lukewarm Kubernetes operator debut, Glasskube founders Philip and Louis shared their launch post-mortem on Hacker News. They discovered that many others were experiencing similar frustrations with the existing package management solutions for Kubernetes. Recognizing this shared pain, a product pivot became inevitable, and the Glasskube package manager began to take shape.

Now, it’s only mid-August of the same year, and we are already closing in on 1.5 million downloads, 2.6k GitHub stars, and 50+ contributors. We've witnessed significant growth in adoption, features, and use cases. As we continue to evolve, we wanted to take a moment to reflect on the work we've accomplished so far. This blog post serves as a comprehensive update, a flag in the road, to share everything you need to know about the current state of Glasskube, what’s on the horizon, and how you can help shape the future of Kubernetes package management.

Welcome to the new frontier of Kubernetes Package Management.

Introduction

Why Kubernetes needs better package management

Since its inception Kubernetes as a container orchestration system has never stopped growing in popularity and its adoption keeps increasing. In recent years, the number of available packages has grown from about 100 to over 800. This growth by all means is very impressive and shows the maturity of the Kubernetes ecosystem, but it has also shined a light on the problems with current package management tools. Developers often struggle with:

Complex and highly bespoke workflows
Time-consuming and many times manual processes
An over-reliance on current package manager tooling that leave a lot to be desired.

These issues can lead to:

More work for operations teams
Non-scalable workflows
Possible security risks

It's clear that Kubernetes needs a way to manage packages that evolve in the way that modern-day Kubernetes clusters are used, that is in a more automated and overall declarative way.

What is Glasskube?

Glasskube is an open-source package manager made for Kubernetes. It aims to make installing, updating, and setting up packages easier and faster. In fact, it can be up to 20 times quicker than tools like Helm or Kustomize.

Glasskube took ideas from easy-to-use package managers like Homebrew and npm. It offers two ways to manage packages:

A graphical user interface (GUI)

A command-line interface (CLI)

glasskube serve # to access the Glasskube UI

Current tools: Helm and Kustomize

Helm and Kustomize are two main tools for managing Kubernetes packages. They help set up and run applications in Kubernetes, but work differently.

Helm uses templates to create Kubernetes yaml files. It packages apps into "charts," which are sets of YAML files. Some of it’s benefits are:

Configuring complex apps
It shines at app installation
Rolling back changes is straightforward
It’s highly adopted

Kustomize takes a different approach and many times works in conjunction with Helm. It lets users:

Set up basic app configurations
Add changes for different environments
Avoid complex templating

Problems with existing tools

While Helm and Kustomize are useful, they can cause issues for developers.

Helm can be tricky:

Hard to learn due to complex templates
Customizing apps for different setups can be confusing
CRD updates are not possible
Updates are clunky
Helm is inherently one-sided, after installation its job is done.

Kustomize has its own problems:

No built-in way to manage packages
Users need other tools to share their setups
Handling complex dependencies can be hard
Less automated than Helm for managing app lifecycles

These issues show why a simpler package manager for Kubernetes is needed. This is where Glasskube comes in, aiming to make the whole process easier.

Care to support us?

If this is the first time you've heard of Glasskube, we are working to build the next-generation Package Manager for Kubernetes.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Glasskube explained

How Glasskube works

Glasskube makes managing Kubernetes packages easier. It offers two ways to use it:

A graphical interface (GUI)
A command-line interface (CLI)

Glasskube uses a central package repository called the Glasskub Hub that:

Aware of package dependencies
Works well with GitOps
Updates packages automatically
Allows for multiple public and private package repositories to be referenced

Probably most importantly, Glasskube has client and server-side components. The server-side components constantly track the health status and desired state of the installed packages. The server-side components are the:

PackageController: communicates with the Kubernetes API to deploy and reconcile packages.
PackageInfoController: reads package manifests from the configured backend package repositories to determine the state of truth of the cluster packages.

Main features

Glasskube has several key features:

Solving common issues

Glasskube directly addressed gaps that other Kubernetes package managers have:

It's easier to use than Helm, which can be hard for new users
It handles dependencies better than Kustomize, which doesn't have built-in package management
It works well with GitOps, making it easier to:
- Roll back changes
- Upgrade apps
- Work together as a team
Package scopes help to be more efficient with package installation and package sharing. Ex. if multiple packages depend on cert-manager, one instance will be enough to serve more than one package.
Have easy access to packages with frontends without having to manually portforward

How to set up Glasskube

What you need to install

Before you start, make sure you have:

A working Kubernetes cluster, minikube is an easy alternative for a quick local setup.
kubectl installed on your computer

Step-by-step installation guide

Here's how to set up Glasskube:

Install Glasskube CLI: Open your terminal and run on MacOs:

brew install glasskube/tap/glasskube

For other installation methods click here.

Bootstrap Glasskube to install the server-side components by running:

glasskube bootstrap

Check the installation Run:

glasskube version

Make sure both Glasskube and the package-operator are installed and running the same version:

glasskube: v0.17.0
package-operator: v0.17.0

Using Glasskube: Key functions

Access the GUI

Glasskube makes it easy to add and remove packages in your Kubernetes cluster:

To add a package:

glasskube serve

Install packages

Any package available in the Glasskube hub is ready to be installed

glasskube install <package-name>

This command will then request confirmation on the package version before installing.

To remove a package:

glasskube uninstall <package-name>

Managing package dependencies

Glasskube takes care of package dependencies for you:

When you install a package, it automatically installs any required dependencies
This saves time and prevents errors from missing components

In the example below upon installing the keptn package, an instance of cert-manager will also be installed if not already present in the cluster.

Handling package configurations

You can easily change or view package settings with Glasskube, note that not all packages have custom value configurations:

In the example below, the caddy-ingress-controller has the automaticHTTPS value exposed, where you can add an email address to enable HTTPS functionality.

Working with GitOps

Glasskube works well in GitOps workflows, letting users control their Kubernetes packages desired state in Git means:

Your Kubernetes package setup stays in sync with your files stored in Git
You can use tools like ArgoCD or Flux to apply the desired package state stored in Git
Changes are easy to track and undo if needed
Cluster setup can be achieved in minutes instead of hours.

We have a fully operational Glasskube GitOps template here. Follow the steps in the README.md file to get started.

Automatic updates with Renovate

Glasskube keeps packages in Glasskube hub always updated to the latest stable versions:

Enable auto-updates or get notified when new versions are available
Shows package information including the current version and available versions for packages using glasskube list
Running glasskube update <packageName> will install the newest available version of the package
Also updating manually through the GUI is also an option.

Adding custom package repositories

You can now add multiple package repositories for Glasskube to read from, this is be useful for:

Companies that have their own private package lists
Teams who want to manage internal packages more easily
It allows following compliance rules for private packages that can’t be exposed publicly

Add a public repository by running:

glasskube repo add <name> <url>

Add private repositories

glasskube repo add <name> <url> --auth (none|basic|bearer)

Glasskube vs. other package managers

How it differs from Helm and Kustomize

Glasskube offers a new way to manage Kubernetes packages, addressing limitations found in older tools like Helm and Kustomize. It offers features such as two-way communication between the package and package manager, enhanced automation, and a more declarative approach to package management. Here's how Glasskube stands out:

Automatic Notifications: Glasskube keeps track of installed versions and can automatically notify users of new installations, if desired.
Seamless CRD Updates: It updates Custom Resource Definitions (CRDs) smoothly, ensuring everything remains in sync.
Full GitOps Integration: Glasskube enables a complete GitOps workflow, which wasn't possible before.
Efficient Control: The additional layer of abstraction through package scopes allows for more efficient and granular control over package configuration and usage.

Business service offerings

Companies that offer commercial tools with self-hosting options often leave money on the table. Custom self-hosting installations and management might not be within your company’s expertise or bandwidth. A Glasskube Native package can be the solution you need. We can collaborate with you to create standardized, customizable, and highly scalable packages tailored for a variety of self-hosted environments, all using Glasskube packaging. This approach ensures a seamless experience for your customers while maximizing your revenue potential.

If you think that Glasskube Native packages could benefit your business, book a free demo call with us. Our team would be delighted to assist you.

What's next for Glasskube

New features coming soon

Glasskube is adding new tools to make package management easier:

These new features will help users manage their Kubernetes packages better.

Wrap-up

Kubernetes package management has been stagnant and unable to keep up with the fast growth and evolution of the general Kubernetes ecosystem. Glasskube aims to make Kubernetes package management easier, more declarative, flexible and far less reliant on rigid templating processes that require multiple tools to get the job done.

As Glasskube grows, we will need help from users and developers to build in the right direction. What frustrates you the most about current Kubernetes package managers? How can we improve?

For teams using Kubernetes, we hope tools like Glasskube will be increasingly adopted and applied in complex environments.

2024 has been an incredible breakout year so far, but we have only just begun.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

The GitOps Kubernetes starter template that gets you set-up in minutes instead of hours

Jake Page — Thu, 08 Aug 2024 15:13:45 +0000

We have all heard off writers looking at a blank page for hours trying to get over writers block. For Kubernetes admins, setting up a brand new Kubernetes cluster can be just as daunting.

Especially when it involves configuring multiple elements like ArgoCD, a PostgreSQL database, monitoring tooling, and custom web apps.

However, what if there was an easier way? Our new GitOps template is designed to make this process straightforward and hassle-free. It offers an easy to set up solution that requires almost no manual steps for a basic yet highly extendable GitOps setup. With this template, you can have your cluster up and running quickly and efficiently, allowing you to focus on what matters most, building and scaling your applications.

If you’re reading this, you likely already appreciate the benefits of GitOps and have a solid understanding of Kubernetes. However, until recently, there hasn’t been an elegant, standardized process for managing the packages installed in your Kubernetes clusters. That’s why we built one.

Let’s put it to the test. With this template, ArgoCD, CloudNativePG and a simple bookmarking web app will be installed using the Glasskube Package Manager and they’ll receive upgrades through a Renovate integration via pull requests. This future-proof setup is easy to maintain and ensures your clusters stay up-to-date effortlessly.

The next logical extension for infrastructure IaC

There are clear frameworks, templates, and standards for provisioning infrastructure across almost all cloud providers, and even templates for provisioning Kubernetes clusters themselves. However, this is often where the smooth, paved Infrastructure as Code (IaC) road turns to gravel. Kubernetes admins are frequently left to their own devices to install, configure, and manage the packages within the Kubernetes cluster due to a lack of standardized and simple package management tooling.

Converting package configurations to code and managing them through GitOps, as you would with internal applications, has proven difficult. We believe the Glasskube package is a significant step toward the much-needed standardization in this area.

What is a Glasskube package?

A Glasskube package is a standardized unit for managing software deployments within Kubernetes clusters using the Glasskube Package Manager. It is defined by a PackageManifest, which contains all necessary information for identifying, configuring and installing a package. This manifest can include a Helm resource or a link to a manifest. Dependencies between packages can be declared, ensuring all required components are present before installation.

Use ArgoCD to deploy Glasskube packages

Helm charts or plain Kubernetes manifests lack the comprehensive framework that Glasskube packages offer for deploying resources declaratively in a reliable, consistent, and easily maintained way. In this demo, we’ll use the GitOps template to bootstrap a fresh Kubernetes cluster with an instance of ArgoCD, which will then be used to deploy all subsequent packages.

Here’s what the cluster will look like.

Template diagram

Template structure

🔗 Link to template

The repository contains:

a bootstrap directory containing the initial/parent Argo Application, and the necessary Glasskube manifests.
a packages directory containing the ArgoCD ,cloudnative-pg and kube-prometheus-stack cluster packages.
an apps folder containing a simple shiori bookmarking web app.
the renovate configuration file.

Glasskube custom resources will only be picked up by ArgoCD after placing the package definition files inside the packages directory.

⚠️ Please do not delete/uninstall the argo-cd package, as this will also remove everything else!

Note that the parent application used to bootstrap (bootstrap/glasskube-application.yaml) will not be synced after the initial setup. If you want to change something about it, you will have to change the application via argo directly.

Cluster setup

Prerequisites

You will have to have access to an empty Kubernetes cluster.

The easiest would be creating a new Minikube cluster with:

minikube start -p gitops

⚠️ Glasskube should not yet be bootstrapped in the cluster.

Install the Glasskube CLI

Make sure to have at least Glasskube version 0.16.0 installed locally. If you don’t, you can simply run:

brew install glasskube/tap/glasskube

How to use the template

Use this repository as your GitOps template

Create a public GitHub repository based on this starter template by clicking “Use this template“. You can move it and/or make it private afterward.

Replace the placeholder repoURL in your local GitOps repository

Replace the default value of repoURL to your repository url:

Line 12 in: bootstrap/glasskube-application.yaml
Lines 11, 16 and 26 in: bootstrap/glasskube/applicationset.yaml

Bootstrap ArgoCD and Glasskube for your Kubernetes cluster:

glasskube bootstrap git --url <your-repo>

The result

As a result, your cluster will be powered with GitOps capabilities by ArgoCD, as well as package management features by Glasskube. Argo manages itself, the Glasskube installation, as well as Glasskube packages — all of which you can now manage GitOps-style with this repo.

Run glasskube serve to open the Glasskube UI and either open the ArgoCD UI there, or with the command glasskube open argo-cd – but of course you can also use the Argo CLI. Follow the ArgoCD docs to get and reset the password to log in.

💡 Note that it might take a couple of minutes to start up ArgoCD, and for the initial GitOps sync to happen.

In this template, for demonstration purposes we’ve also installed the cloudnative-pg and kube-prometheus-stackclusterpackages and a bookmarking application (shiori), which is making use of cloudnative-pg.

Managing your cluster

Both CLI and UI offer features to manage your cluster following GitOps best practices.

Using the CLI

The relevant CLI commands offer the flags --dry-run and -o yaml which output the yaml object code which should then be pushed to your repository to be deployed.

Through the UI

The UI, when installed with the above glasskube bootstrap git command, will also output the yaml objects which you can copy to use in your git repo, instead of applying your changes directly.

Installing packages

To install a ClusterPackage, e.g. cert-manager, use the install command like this:

glasskube install cert-manager --dry-run -o yaml --yes > cert-manager.yaml

Instead of directly installing the ClusterPackage, this will write the ClusterPackage custom resource to the cert-manager.yaml file, which can now be put into a new directory packages/cert-manager/ in the git repository. Once pushed to the repo, ArgoCD will pick up the changes after at most 5 minutes, create the ArgoCD Application wrapping the Glasskube ClusterPackage. After that, the Glasskube operator will pick up the ClusterPackage and finally install it in the cluster.

Similarly, when using the Glasskube UI, one can generate the ClusterPackage resource by using the “Show YAML” button on the page of the ClusterPackage.

Updating packages

There are two options handling package version updates:

Manually

Using the glasskube update --dry-run -o yaml command, or the corresponding button on the Glasskube UI. And then pushing to your repo as seen before.

The downside of doing it that way, is that someone has to manually execute the command, even though checking for updates and preparing the updates to the git repository as an automatable task.

Automatically with Renovate

Once Renovate is integrated to track your GitOps repo, it will look for Glasskube packages and compare their versions to the official package repositories. When new versions are available, it will automatically open a PR. Once merged, you’ll be running the latest versions of your packages.

Uninstalling packages

To uninstall a package or a ClusterPackage, simply remove the custom resource from the git repository.

Updating Glasskube

When a new Glasskube version is available, the manifests have to be updated. To update the Glasskube manifests in your git repo, run the following command:

glasskube bootstrap --dry-run -o yaml --force > bootstrap/glasskube/glasskube.yaml

After reviewing and merging those changes the update will be picked up by ArgoCD. The --force flag is necessary for the command to continue manifest validation, even though failures occur.

Working with Apps

This template also contains a demo application: a bookmark manager called shiori.
Its manifests are defined in apps/shiori, which is a pattern you can follow for your own custom applications.

In a minikube environment, two manual steps are required to access the application (for more information consult the
minikube docs):

Run minikube addons enable ingress -p gitops.
Run minikube ip -p gitops and add the line <your-IP> my-shiori.example to your /etc/hosts file.

After that you will be able to access the application via http://my-shiori.example in your browser.
The default login credentials are shiori / gopher – for more information check the shiori docs.

In general, you can use the apps directory to deploy such custom applications into your cluster. Any subdirectory will be
picked up by ArgoCD and grouped as an Application.

Monitoring with `kube-prometheus-stack`

This template also installs the kube-prometheus-stack clusterpackage, which is an easy way to get started with monitoring your cluster. You can open Grafana with glasskube open kube-prometheus-stack. It does not come preconfigured in this example, but you can easily add a nice postgres dashboard and observe the metrics of the database while you are working with the bookmarking application.

Setting up a postgres dashboard

We are going to make use of the cloudnativepg dashboard.
Import it by opening the dashboard-import page, pasting
https://grafana.com/grafana/dashboards/20417-cloudnativepg/
into the first textfield, and pressing "Load". Use the "Prometheus" data source on the following screen and finish the import process.

Of course monitoring your experimental minikube cluster is a bit of an overkill, but this is simply to demonstrate how these kind of cluster administration tasks can be integrated into this gitops stack.

Template setup walkthrough

Upcoming Features

Private Repo Support

We are aware that GitOps repositories should not be public, but for simplicity we omitted this feature in the first version of the new GitOps-bootstrap command. Supporting private repos with authentication of course has high priority for the upcoming releases. We will also replace the repoURL automatically, such that you don’t need to this step manually when setting up the repo.

Improved Renovate Integration

As described above, the renovate integration currently is regex-based, and it does not consider dependencies yet. However, we don’t see these shortcomings as a blocker and recommend to try out the renovate integration in the Glasskube/Argo Gitops setup.

Improved Dependency Resolution

Installing packages with dependencies is not 100% GitOps-compatible yet, as the dependencies will be created by the operator. Consider this: To install a ClusterPackage <P> that has a dependency on D, one would do glasskube install <P> --dry-run -o yaml, which would output the ClusterPackage custom resource for <P>. However, the dependency D will only be resolved at reconciliation time by the package operator, and will therefore not be represented in the git repository at all.

A temporary workaround would be to have a closer look at the output of the install command, which also shows the dependencies which will be installed and in which version. One could then manually add the required packages custom resources to the git repo as well. However, this will be tackled in a future version to make the user experience better, see this issue.

Summary

With this template repository and guide we show how Glasskube can easily be set up in a ArgoCD powered Gitops environment, and how efficient package management is possible with this stack. Additionally we install a web application to show how custom applications can make use of the Gitops setup and Glasskube packages.

This is a first concept with some minor shortcomings, but we will continue to improve GitOps support.

Feedback

We love feedback! Whether you are just starting out or you are a seasoned professional, we'd like to hear your thoughts, inputs and questions regarding this starter template and corresponding guide here, in the glasskube/glasskube repo or on our Discord. Thanks!

Why building an IDP for Series A companies doesn't make sense

Jake Page — Wed, 07 Aug 2024 10:21:01 +0000

CTO of a Series A company? Before reading on, just give yourself a quick pat on the back. What you and your team have done so far is no easy task, so fair play to you.

Realistically though, the work is far from done. Many scaling and growth decisions lay ahead and probably more questions than clear convictions.

You might be tempted to preemptively adopt tooling and processes, positioning your engineering team for the potential growth to come. In this piece we’ll explain why you are more likely to succeed by doubling down on what brought you to this point over adopting new processes or fancy Internal Developer platforms.

Setting the right foundations

Series A companies are exciting environments to be in, usually a reasonable influx of new faces join the company, processes are increasingly standardised and overall, there is a progressive company-wide maturity that is starting to take hold. The instinct to mature the engineering org is understandable but nonetheless ill placed.

No level of maturity or sophisticated tooling can compensate for a lack of Product Market Fit (PMF). Validating PMF across markets should be top of mind at this stage above all else. Elusive as it may be, your best chance of achieving it is staying agile, delivering rapidly and maintaining tight feedback loops.

The idea of planning for future scalability might be appealing, especially with the buzz around platform engineering and the availability of ready-made internal developer platform (IDP) solutions. However, it’s crucial to understand why these options might not be the right choice for you right now. Let’s delve into the reasons why an IDP isn’t the change you think you need.

What is an IDP?

An IDP is a self-service layer build on top of your organisation’s infrastructure, that enables your developer teams to deploy and manage applications without the need to have in-depth infrastructure knowledge or relying on the help of traditional infrastructure engineers. It aims to provide consistency, automation, and speed by standardizing workflows and environments as the organization grows.

Think of an IDP as an internal product maintained by DevOps or Platform engineers, who essentially serve their own customer base, your development teams. To understand why adopting an IDP might be counterproductive for a Series A company, it’s essential to understand why IDPs emerged in the first place.

As companies expand, so do their engineering teams. In large enterprises, maintaining code quality, keeping up with infrastructure requirements and sustaining deployment velocity is hard if operations teams can’t keep up. IDPs were designed to remove reliance on traditional operations teams, to rely on a platform that actively evolves to fit their needs.

A few assumptions are important to point out here:

You have the resources to maintain a completely new product (the IDP). You probably don’t.

You have already scaled the engineering org and now need to optimise it. You probably haven’t and don’t need to.

What should Series A companies care about?

Before going on about the types of tools and processes your shouldn’t care about, let’s take a more proactive approach and focus on things that will actually move the needle.

The right team

You need a group of people who can run themselves, have the expertise to automate infrastructure related tasks and are open to working in flexible and not overly standardised structures. They should be the ones who propose changes and implement them as needed.

Finding and validating PMF

By the time you are at a Series A level you have surely caught the scent of where you might find PMF. Do not let up, ship fast and communicate intensely with your target audience to validate your market assumptions. There is no reward in finding a gap in the market alone, you have to occupy it. Maintain development velocity to increasingly occupy the gap you have identified.

Staying agile and flexible

The engineering team, like all teams within the company, should be unequivocally aligned with one primary objective: enabling the business. The engineering team sits at the centre of a short but tightly knit feedback loop which includes sales, feature ideation, feature development, deployment, and gathering feedback. Ideally, the sales team is working flat out, and customer service is maintaining close contact with customer. For the engineering team to keep it’s end of the bargain, it has to stay flexible and agile, ready to respond quickly the other teams inputs.

Focusing on customer facing products only

Even though we’ve mentioned this before, it’s important to emphasise that an IDP is a fully-fledged product that needs to be created, maintained, and evolved to meet the changing needs of internal developers over time. Focusing on an internal product like this can detract from the attention and resources needed for your customer-facing products, which ultimately bring you in closer contact with PMF and customer satisfaction.

Building the right culture

This is a point that should not be overlooked. Regardless of your company’s stage, evolution is always and ongoing dynamic. A culture of alignment, healthy communication principles, and a focus on shared goals is the bedrock that must serve as a base for the discussions about architecture, tooling, and process definitions which enable said evolution. Collective buy-in can only happen if a meaningful shared culture exists.

What should a Series A CTO not be thinking about?

It’s not un-common to hear people report having learned more from their bad managers than from their good ones. Why might that be? Because knowing what not to do can sometimes be more important the the opposite.

Don’t maintaining internal products

Any time spend not addressing requests from your product and sales teams and hopefully your customers is time wasted.

Any time spend that doesn’t involve delivering a better experience than your competitors is a mistake.

Avoid large engineering teams

Just because the rest of the company is growing doesn’t mean the engineering team has to expand at the same rate. Those who praise the magic that can be harnessed by keeping engineering teams small are on to something.

At this stage, as you may be bringing on your first DevOps engineers, ensure they integrate into a compact, developer-focused team with a single leader to avoid unnecessary managerial layers, allowing for open communication and quick decisions to be made.

Forget about perfection

It’s not news to you that shipping quickly and continuously improving over time is the best way to build a product that eventually wins market share and meets customer expectations. Why would scaling engineering teams be any different? Trying to implement an Internal Developer Platform too soon runs contrary to those principles. There will be a time when you will build one out but let that process happen iteratively in a gradual manner. You have others things to worry about in the meantime.

You are on the right track

Reaching a Series A funding round is not the end goal for anybody, you have a lot of room for growth. PMF is more than likely not locked down and occasional engineering bottlenecks should be dealt with on a case by case basis. Now is the time to double down on the strengths and keep on improving you capacity to identify your customer and increase user satisfaction.

There will be a time to look inwards and improve internal engineering processes, and IDP might make sense in the future, but at this moment in time your limited engineering output should be laser focused on staying lean.

Implementing internal products that don’t directly enable the business too soon have the potential for dire consequences. Approximately 35% of series A funded startups fail before Series B, a lot is at stake.

No need to worry though, you have made it this far, don’t give into platform engineering FOMO, put you head down and keep doing what you’ve been doing, because it’s working.

We would love to hear from you, what are your thoughts? when is an organization really ready for an IDP?

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

20 Life hacks for DevOps Engineers

Jake Page — Tue, 23 Jul 2024 09:29:37 +0000

Tricks of the trade, hacks, trade secrets, cheat sheets, best practices, call them what you will. Every industry has them, and anyone who sticks around long enough builds an arsenal of techniques and finely tuned tools to excel at their job.

Some things simply take time to master. My dad, a retired builder, could tile a medium-sized bathroom in under an astonishing three hours, while it would take me a full day just to do the grouting afterwards. Experience is key for certain skills, but there are also tips that don’t require years of practice to acquire.

DevOps is no different. There are no shortcuts to becoming a god-tier DevOps savant, put the years in and you might just get there. Having said that, there are some tricks of the trade, life hacks and useful tooling that are sure to give you an instant boost in productivity.

Here is my non-comprehensive list of life hacks guaranteed to make any DevOps engineer’s life easier.

The list is broken down into:

Tooling 🧰
Skills 🤸
Habits 🔁
Scripts, configs and extensions 💻

Tooling: 🧰

Did you know that if you really want to get somebodies attention in Germany, sending a fax is your best bet?
Also, in Japan up until this year, floppy disks were still in use by government institutions.

Knowing which context you find yourself in is essential to then choose the best tools for the job. Even though it’s important not to obsess over having the best, newest, or shiniest tool on the market. The following tools can be real game-changers for DevOps engineers.

"You don’t need to be the sharpest tool in the shed to use the sharpest tools in the shed" - Anonymous (I might have made it up)

1. k9s

K9s is a terminal-based UI for interacting with your Kubernetes clusters. It takes very little time to get accustomed to navigating, observing, and managing your live applications through the tool. And once you do, you might never go back. K9s continually monitors Kubernetes for changes and offers many useful commands to interact with your observed resources.

Link to instal.

2. tmux

tmux is a powerful terminal multiplexer that enhances productivity by allowing session persistence, window and pane management, and customization through key bindings and configuration files. It supports scripting for automation, facilitates collaboration with shared sessions, and integrates well with various shells and tools.

3. Glasskube

Glasskube is an Open Source package manager for Kubernetes. It makes deploying, updating, and configuring packages on Kubernetes 20 times faster than tools like Helm or Kustomize. Inspired by the simplicity of Homebrew and npm. You can decide if you want to use the Glasskube UI, CLI, or directly deploy packages via GitOps.

Care to support us?

If this is the first time you've heard of Glasskube, we are working to build the next generation Package Manager for Kubernetes.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

4. ripgrep

Riggrep is a powerful search tool known for its speed, flexibility, and user-friendly output. It quickly processes large codebases using advanced algorithms, supports a wide range of search patterns, and presents clear, highlighted results. Riggrep integrates well with other tools, is cross-platform, and customizable.

5. Firefox containers for multi cloud account access

Firefox Multi-Account Containers is a such an underestimated browser extension for it's utility. It helps manage online activity by separating websites into different containers or tabs, preventing sessions tracking across sites. It’s most useful feature is that it allows users to log into multiple accounts simultaneously within the same browser. By isolating sessions through cookie separation, it protects personal data and improves the overall browsing experience. Have multiple AWS accounts? Not an issues, you can log in to all of them from the same browser window.

6. VPA

Vertical Pod Autoscaler (VPA) frees users from the necessity of setting up-to-date resource limits and requests for the containers in their pods. When configured, it will set the requests automatically based on usage and there after allow proper scheduling onto nodes so that appropriate resource amounts are available for each pod.

Install it here.

Example config:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       my-app
  updatePolicy:
    updateMode: "Auto"

7. Kctx and Kubens

Is just hand’s down the most useful CLI tool for Kubernetes context and Namespace switching.

Install it here

8. ChatGPT for guidance

Use ChatGPT as if it were a senior member on your team who is never busy and always happy to answer all of your questions. Make sure your intention is deeper understanding and not blind task resolution.

Prime ChatGPT to be a senior team member with a prompt like this:

You will act as a Senior DevOps Engineer to provide life hacks and tips on how to excel in the field of DevOps. You will also be ready to help junior team members with any questions they might have. Please include practical advice, recommended tools, and best practices for managing infrastructure and continuous integration/continuous deployment (CI/CD) pipelines. Write the output using my communication style, which is clear, concise, and practical. Here are examples of my communication style:

"Focus on automating repetitive tasks to save time and reduce errors."

"Leverage tools like Docker and Kubernetes for containerization and orchestration."

"Always monitor system performance and be proactive in identifying potential issues."

"When mentoring juniors, be patient and explain concepts in simple terms."

Here are some questions you can ask ChatGPT to further fine tune the prompt:

Do you want the advice and explanations to cater more to beginner-level juniors or those with some experience in DevOps?
Could you provide more detailed examples of your communication style, especially in scenarios where you explain complex concepts to juniors?
Are there specific challenges or areas of focus within DevOps (e.g., automation, monitoring, security) that you want to prioritize for the advice and junior support?

Skills: 🤸

It's no surprise that skills can't be absorbed instantly, they require time and effort. In the ever-evolving tech world, determining which skills to cultivate can be confusing. However, as DevOps engineers, you can't go wrong with mastering scripting and prioritizing documentation.

“Build your skills not your resume.” — Sheryl Sandberg

9. Scripting

Scripting is like a swiss army knife for DevOps engineers because it automates repetitive tasks, provides essential glue between processes, and ensures consistency across environments. Learning and practicing scripting involves familiarizing yourself with tools like Makefiles, regular expressions (regex) for efficient text processing, and Bash scripting for powerful command-line operations.

Don't feel like you have to master scripting before implementing it at work. Look for manual steps in your current software or infrastructure delivery processes and try to script automations that make sense. Leverage an LLM for help if needed.

10. Documentation

Write everything down. It’s the best way to take care of your future you.

Try out a few note taking solutions here are a few:

It doesn’t matter which one you choose as long as you only choose one and you stick to it.

🤔 Don’t fall into the trap of spending too much time organising and optimising your notes. Notes don’t have to be perfect only functional. Spending too much time on note maintenance is meta-work.

Habits: 🔁

You will never get far if you rely solely on pure motivation to get things done and improve. Well-defined and consistent habits are the glue that keep you growing and effective, even when your motivation wanes. Habits seamlessly integrate tools and skills, ensuring you are effective at your job.

"Motivation is what gets you started. Habits is what keeps you going." - Jim Ryun

11. Don’t write to-do lists, block time.

I’ve been heavily influenced by Cal Newport’s writing on Deep Work and I’m quite convinced that efforts to preserve a certain amount of time per week dedicated to uninterrupted deep work is crucial for an individual who wants to contributor meaningfully to a team.

To-do list on their own are just wish lists. Once you plot them on a calendar, that’s when you have a plan.

It’s important to point out that I know very few people who execute on their time block plans with 100% fidelity every day of the week. Use them as north stars, schedule rest time when you need it. Update the list throughout the day even. But at least hold yourself to a plan.

12. Reciprocal meeting blocks

Unless you are your own boss, you’re most likely not 100% in charge of the meetings and commitments you might be obliged to participate in. Reciprocal meeting blocks is a way to counteract unexpected time commitments that just pop up in your calendar.
The idea here to reserve an equivalent deep work block every time a new meeting is added to you calendar. That way you can stay flexible and relatively available without having to compromise on your weekly deep work quota.

13. Have a shutdown routine

Especially useful if you are a remote worker. A shutdown routine is a series of questions and steps your run through to finish your working day. Ideally once you complete the checklist you should be able to forget about your work until the next day.

What I track is the following:

Did I exercise?
Did I address all of the miscellaneous tasks I had?
Do I have any open conversations?
Do I need to push any tasks to the next day?
Did I write a diary entry? (not work related but I like to get it done before I leave the desk)
How many deep hours did I do?
Did I check the metrics I’m tracking one last time before I close the laptop for the day?

14. Take notes during meetings

Taking notes during meetings and sharing them afterward should ideally be a common practice in your organization. If it isn't, this is a perfect opportunity for you to start. Not only does it ensure that no important details slip through your fingers, but it also provides a great service to your team.

15. Test run outages

Test run outages are essential for DevOps engineers to prepare for real incidents. You have to know exactly how to swiftly and efficiently connect to your clusters or VMs before an unexpected 1 a.m. alert for example, you can’t be caught off guard.

Familiarize yourself with moving files, retrieving container logs, and other critical tasks. Setting up SSH keys, kubeconfigs, and other access tools in advance will save valuable time and reduce stress during actual outages. Proactively testing and optimizing these processes ensures that you are ready to respond effectively to any disruption.

Scripts, configs and extensions: 💻

If you have to do something more than once, automate it. Why write out a full command when you can save time by using an alias? Over a long career, it might be hard to calculate exactly how much time is saved by typing "k" instead of "kubectl." One thing is for sure: it's a lot, and it's well worth it.

“You’re either the one that creates the automation or you’re getting automated.” — Tom Preston-Werner

16. Use helpful aliases

Don’t waste time typing out the full commands you write every day.

Here is a small snippet of a few of my configured aliases:

k=kubectl
kctx='kubectl ctx'
kgp='kubectl get pods'
kns='kubectl ns'
l='ls -lah'
la='ls -lAh'
ll='ls -lh'
ls='ls -G'
lsa='ls -lah'
md='mkdir -p'
rd=rmdir
run-help=man

17. Efficiently cleaning up finished jobs with TTL controller

You can specify the lifespan of finished jobs or pods before they are automatically removed by setting the .spec.ttlSecondsAfterFinished field. If you working in a job heavy environment finished jobs can quickly accumulate and become quite resource heavy.

Example config:

apiVersion: batch/v1
kind: Job
metadata:
  name: test-ttl-job
spec:
  ttlSecondsAfterFinished: 100
  ...

18. Git script to keep synced with the upstream

git remote add upstream <upstream-url>
git fetch upstream
git rebase upstream/main
git push --force-with-lease

19. Kubectl auto complete

Kubectl autocompletion lets you create an alias for kubectl. This feature saves time by reducing the need for cheat sheets and is especially useful for managing Kubernetes clusters. It's also recommended for the CKA exam due to its time efficiency.

Set up for Linux:

# install bash-completion
sudo apt-get install bash-completion

# Add the completion script to your .bashrc file
echo 'source <(kubectl completion bash)' >>~/.bashrc

# Apply changes
source ~/.bashrc

Check out other installation methods here.

20. Visual Studio Code Remote – SSH

The Remote - SSH extension lets you use any SSH-enabled remote machine for development, making it easier to develop on the same OS you deploy to, use powerful hardware, switch between environments, and debug applications remotely.

Install it here.

Final thoughts

There’s no magic formula for becoming a top 1% DevOps engineer. Like in most professions, it’s a matter of time, dedication, and experience that will gradually shape you into a highly effective professional. Over time, you'll get much better at recognizing patterns, recalling past situations, and finding quick solutions to recurring issues. So, don’t expect any single life hack from this list to instantly earn you a 50% raise and a promotion.

However, if you consistently focus on refining your tools, honing your skills, refusing to break good habits, and implementing smart automations, you’ll be well on your way to rapidly evolving and surpassing your current self. And who knows, maybe that promotion is just around the corner.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Simplify Kubernetes Monitoring: Kube-prometheus-stack Made Easy with Glasskube

Jake Page — Mon, 15 Jul 2024 10:18:11 +0000

What do we, as developers and engineers, value most above all else? The answer is simple: our time.

Tools that deliver value in the shortest amount of time have the highest chance of user adoption, it's as simple as that.

What else do most engineers value? Beautiful and data-rich dashboards.

Prometheus and Grafana are open-source, community-backed solutions with stellar reputations. They bring immense value by fetching and storing metrics while enabling the creation of dashboards that are not only useful but also easy on the eyes.

The uncomfortable truth is that anyone who has ever set up Prometheus alongside Grafana as their environment's monitoring stack from scratch has probably felt the frustration of not getting value especially quickly. Metric exporter configuration, dashboard widget customisation and deciding what to monitor and alert on in the first place takes time.

That's why Kube-Prometheus-Stack was created. It installs a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules, providing an easy-to-operate, end-to-end Kubernetes cluster monitoring solution with Prometheus using the Prometheus Operator.

This sounds like good news, and it is, but the stack is bundled in a Helm chart, and just the values.yaml file has over 4000 lines. Configuring and maintaining the Helm chart isn’t necessarily straightforward or “fun.”

With so many configuration options, we must be getting something good right? Well yeah, we are, by deploying kube-prometheus-stack we get all of this right out of the box:

Top Layer:

User: Interacts with Grafana and Kubernetes API.

Visualization and Alerting Layer:

Grafana: Connects to Prometheus for data visualization.
Alertmanager: Connected to Prometheus for alert management.
Prometheus Server: Central component collecting and storing metrics.
ServiceMonitors & PodMonitors: Define how Prometheus scrapes metrics.
Prometheus Rules: Includes both alerting and recording rules.

Exporters Layer:

Node Exporter: Collects node-level metrics.
Kube State Metrics: Collects metrics from Kubernetes API objects.
Other Exporters: Additional exporters for various applications and services.

Kubernetes Cluster:

Kubernetes Nodes: Running applications and system components.
Applications: Monitored by the Kube Prometheus Stack.

Luckily, Glasskube now supports the Kube-Prometheus-Stack. Package configuration, lifecycle management, and installation can be done in record time.

In this blog post, we will explore the steps to configure and install the Kube-Prometheus-Stack using Glasskube, wasting no unnecessary time wrestling with never-ending values files and getting you working dashboards and alerts quicker than ever before.

Requirements:

Access to a Kubernetes cluster (Minikube will be fine)
Glasskube installed
An extra screen for all the cool dashboards you are going to want to look at all the time. 🤣

Before we begin

For us at Glasskube crafting great content is as important as building great software. If this is the first time you've heard of us, we are working to build the next generation Package Manager for Kubernetes.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Create a cluster

Install Minikube then run:

minikube start

Check your installation by running:

minukube status

Desired output:

➜  ~ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Install Glasskube

If you already installed glasskube you can skip this step.
If not, glasskube can easily be installed by following your distribution's specific instructions.

MacOs:

brew install glasskube/tap/glasskube

Linux (Ubuntu/Debian)

curl -LO https://releases.dl.glasskube.dev/glasskube_v0.4.0_amd64.deb
sudo dpkg -i glasskube_v0.4.0_amd64.deb

More installation guides here

After installing Glasskube on your local machine, make sure to install the necessary components in your Kubernetes cluster by running glasskube bootstrap. For more information, check out our bootstrap guide.

Once Glasskube has been installed access the UI with:

glasskube serve

Navigate to http://localhost:8580/ to access it.

Kube-prometheus-stack installation

Installation can be done via the CLI, UI or even through a YAML package definition file. Since we will customize the deployment we will use the UI for this example.

Package Customization

Glasskube offers a series of customisations that we can be tweaked and adjusted from the CLI or GUI, saving you from having to render and configure the values.yaml file directly.

Let’s take them one by one.

Enable Alertmanager 🚨

We want Alertmanager to be enabled so we can leverage the metrics prometheus exposes to create helpful alerts.

Grafana Domain 📊

We will leave this empty for this demo since we would need to deploy an ingress controller to our cluster to handle the ingress object associated with the Grafana service. We could use Ingress-nginx or Caddy-ingress which are also supported by Glasskube for this.

Glasskube will automatically port-forward the Grafana pod so we can access the dashboard via the Open button.

Node Exporter host network 💽

Let’s also enable this to export node level metrics like memory and node level CPU usage.

Prometheus retention 📅

This is a duration in days for how long we want to persist the collected metrics.

Prometheus storage size

The amount of storage requests we consider the package will need.

Parameter input methods 🗄️

Glasskube allows for various methods of parameter input:

From a Kubernetes Secret
From ConfigMap
Value from Package Configuration
Via the UI

By choosing to inject data via Kubernetes Secrets, ConfigMaps and Package configuration we can maintain simplicity without compromising security.

Here is the example of how we would reference a specific configMap we have already created and deployed to our cluster.

💡 If you're using Kube-prometheus-stack and considering Glasskube for package lifecycle management but need support for specific key parameter customizations, please open an issue on GitHub with your use case. We'll do our best to expand the parameter list accordingly.

Install via Glasskkube

Once the configuration section is complete, install kube-prometheus-stack.

Upon installation you can see that the kube-prometheus-stack namespace has been created. And a series of pods have been deployed, including the grafana dashboard, the prometheus operator and the kube state metrics pods too.

Access the dashboards

In next weeks blog post we will access the dashboard via a custom dedicated Grafana URL

Hit the Open button or if you want to access Grafana on a different port you can simply port-forward the pod, which will map the exposed Grafana port to a port on your localhost. I've arbitrarily chosen to port-forward to localhost 52222 since it's available.

kubectl port-forward POD_NAME 52222:3000

Head over to http://localhost:52222/ and you will then be greated by the Granfana login page. To find your credentials which are stored in a Kubernetes secret that was generated as part of the deployed stack, run:

kubectl get secret kube-prometheus-stack-kube-prometheus-stack-grafana  -o go-template='
{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

Which will output something like:

admin-password: prom-operator
admin-user: admin
ldap-toml:

Upon access you we be greeted by a long list of powerful pre-configured Grafana dashboards which are already showing local cluster metrics:

Easily access CPU usage information

Here is a segment of the nifty CoreDNS dashboard that also comes preconfigured

Alerting

We already get many useful alerts created for us right out of the box.

In this snippet you can see that some of the preconfigured alerts are already firing: ↘️

If you want to be notified in via email, PagerDuty or any number of third party supported you will just need to add your contact points of preference and then add them as destination inside custom notification policies.

The Kube-prometheus-stack offers tremendous "out-of-the-box" value for Kubernetes cluster monitoring, eliminating the need to start from scratch. It bundles essential components for metrics exposure, extraction, alerting, and visualization, helping you establish a robust monitoring posture from the get-go. With official support from Glasskube, managing and updating a comprehensive, best practice-compliant monitoring stack has never been easier.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Log and trace management made easy. Quickwit Integration via Glasskube

Jake Page — Mon, 01 Jul 2024 12:14:52 +0000

Distributed application troubleshooting can be a nightmare. Unless you have the budget for expensive proprietary monitoring SaaS solutions or the expertise to run and maintain an complex ELK stack you might feel as if you are stuck in a cave without a flashlight.

Luckily, viable open-source alternatives like Quickwit are here to come to the rescue. By weaving together existing tooling for log and trace ingesting as well as pairing well with dashboard and visualisation tools such as Grafana and Jaeger. And sandwiching powerful indexing storage and search capabilities in between. Even if the tool sounds new, it won’t be for long.

We recently integrated Quickwit with Glasskube and it’s available to be easily deployed to your cluster. I spoke directly with Quickwit co-founder François Massot to get the insider scoop, and to learn how the tool works. Let's dive in!

But what is Quickwit exactly? 🤷

Quickwit is a cloud-native search engine that emerged with the goal of creating an open-source alternative to expensive monitoring software like Datadog and Splunk. Along the way, they’ve also developed and open-sourced several components, including ChitChat (cluster membership protocol), mrecordlog (WAL), Whichlang (fast language detection), witty actors (actor framework), and bitpacking (SIMD algorithms for integer compression).

Quickwit, with its robust Elasticsearch-compatible API, integrates well with tooling from the OSS ecosystem, such as Grafana, Jaeger, and OpenTelemetry. Users are successfully deploying Quickwit at scale, with hundreds of nodes and hundreds of terabytes of data ingested daily, all while enjoying significant cost reductions and how thanks to Glasskube to can get up and running in no time.

Quickwit excels in handling logs, traces, security data, and append-only datasets, with plans to support metrics soon. A key feature is the usage of object storage for the indexed data, which simplifies cluster management, cuts infrastructure costs, and enhances reliability. Multiple storage options are available such as local disk, Amazon S3, Azure Blob storage or Garage, an OSS distributed object storage, are available.

Questions for the co-founder François Massot 🙋

What are the benefits of using external Object Storage as opposed to node attached storage?

There are a lot of benefits! From the beginning, we chose to decouple compute and storage to make our search engine scalable, reliable, and very cost-efficient. If you want to remember one thing distinguishing Quickwit from traditional search engines, this is decoupled storage and computing.

Firstly, it provides elasticity, allowing us to scale storage and compute resources independently, which is ideal for cloud environments. Secondly, it’s cost-efficient, as object storage like S3 is cheaper than traditional disk storage, especially for large volumes of log data. And you don’t need to replicate your index data; this is done by the object storage layer. Additionally, it ensures high durability and availability, reducing the risk of data loss. Last, but not least, it simplifies cluster management as most of Quickiwt’s components are stateless.

Performance Comparison: Is Quickwit Faster than Elasticsearch?

It depends!

On indexing, Quickwit is generally twice as fast as Elasticsearch and uses less CPU. Our users, like Binance, report a reduction of 80% of CPU usage at indexing!

The story is different regarding querying, as Elasticsearch has all its data on a local disk, typically SSD, and Quickwit has its indexed data in very slow object storage. In this case, you can expect query time to be lower. But Quickwit's main goal is to be sub-second queries, which is perfectly fine in the observability/security domains. If we look at this indicator, Quickwit is on par with Elasticsearch and is even faster for demanding analytics queries, whereas the data is stored on object storage!

What's in store for quickwit in the future?

We have a very ambitious roadmap! Here are the key features that will be added in the following 12 months:

Distributed ingest (July 2024): High-throughput indexing on tens of thousands of indexes.
OpenSearch Dashboard support (Q3 2024): This will enable OpenSearch users to migrate seamlessly to Quickwit with their existing dashboards.
Metrics support (Q4 2024): New storage engine optimized for time series data.
Distributed SQL engine (Q1 2025): Distributed SQL engine for analytics on top of Apache Arrow, Datafusion, and Ballista.
Pipe-based query language (Q2 2025): Introduction of a flexible and powerful query language similar to SPL (Splunk Query Language)

Use cases

Log management 🪵

Quickwit is built from the ground up to efficiently index unstructured data, and search it effortlessly on cloud storage. Moreover, Quickwit supports OpenTelemetry gRPC and HTTP (protobuf only) protocols out of the box and provides a REST API ready to ingest any JSON formatted logs. This makes Quickwit a perfect fit for logs!

Distributed tracing 📊

Distributed Tracing involves monitoring application requests as they traverse various services like frontend, backend, and databases. It's instrumental for understanding application behavior and diagnosing performance issues.

Additionally, Quickwit seamlessly integrates with OpenTelemetry using gRPC and HTTP protocols (protobuf only), as well as Jaeger's gRPC API (SpanReader only). This means you can store traces in Quickwit and effortlessly query them using Jaeger's UI.

Key features 🔑

Full-text search and aggregation queries
Elasticsearch query language support
Sub-second search on cloud storage (Amazon S3, Azure Blob Storage, …)
Decoupled compute and storage, stateless indexers & searchers
Schemaless or strict schema indexing
Schemaless analytics
Grafana data source
Jaeger-native
OTEL-native for logs and traces
Kubernetes ready via Glasskube
RESTful API

Installation guide 🦮

Prerequisites

Access to a Kubernetes cluster (you can easily create a local cluster by using Minikube or Kind)
kubectl isn't strictly speaking a dependency for installing packages via glasskube, but it is the recommended way to interact with the cluster. Therefore, it is highly recommended. Installation instructions are available for macOS, Linux and Windows.

Install Glasskube

If you already installed glasskube you can skip this step.
If not, glasskube can easily be installed by following your distribution's specific instructions.

For this demo I'll be using a MacOs distribution:

brew install glasskube/tap/glasskube # install the glasskube cli
minikube start # start a minikube Kubernetes cluster
glasskube bootstrap # install glasskube on the kind cluster

For more installation guides, find them here.

Once Glasskube has been installed access via the UI with:

glasskube serve

The dashboard will open up on http://localhost:8580/.

Creating an S3-Compatible Bucket

Before installing Quickwit, you'll need to create an object storage bucket to hold your Quickwit indexes. You can use use your choice of Cloud provider such as Scaleway, AWS S3 or MinIO. Refer to our official Quickwit documentation for storage configuration details.

Here I will be creating an AWS S3 bucket to store the Quickwit indexes.

Steps:

Navigate to the AWS management console and create a new S3 bucket.
In IAM generate an API key, with S3 permissions, save the 'Access Key Id' and 'Secret Key', we will need them shortly.

Deploy Quickwit

From the Glasskube dashboard, find the Quickwit pacakge and add your custom configuration parameters.

defaultIndexRootUri: for this demo it's s3://quickwit-indexes.
metastoreUri: we won't use PostgreSQL so let's pick the same value we used for defaultIndexRootUri.
s3AccessKeyId: the "Access Key Id" from AWS we generated before.
s3Endpoint: Custom endpoint for use with S3-compatible providers. Not needed for S3 configuration.
s3Flavor: we are using the default empty value for genuine S3-compatible object storage.
s3Region: US-east-1 in my case.
s3SecretAccessKey: the "Secret Key" from AWS we generated before.

Here you can find the official Quickwit documentation for parameter completion.

It's also possible to install and configure Quickwit using the Glasskube CLI by running:

glasskube install quickwit

Once installed, you can see that a quickwit namespace has been created:

default
flux-system
glasskube-system
kube-node-lease
kube-public
kube-system
kubernetes-dashboard
quickwit

Now, check to see if the pods are running

NAME                                               READY   STATUS    RESTARTS      AGE
quickwit-quickwit-control-plane-86bd9955f7-bwm2r   1/1     Running   1 (27m ago)   29m
quickwit-quickwit-indexer-0                        1/1     Running   1 (27m ago)   29m
quickwit-quickwit-janitor-9479697ff-x4x2c          1/1     Running   1 (27m ago)   29m
quickwit-quickwit-metastore-56ff74df9f-k6d2g       1/1     Running   0             29m
quickwit-quickwit-searcher-0                       1/1     Running   1 (27m ago)   29m
quickwit-quickwit-searcher-1                       1/1     Running   0             27m
quickwit-quickwit-searcher-2                       1/1     Running   0             27m

We can try to access to the Quickwit UI using the following command:

$ kubectl -n quickwit port-forward pod/quickwit-quickwit-searcher-0 7280

Head over to http://localhost:7280. And you should be ready to go!

Create your first index

Before adding documents to Quickwit, you need to create an index configured with a YAML config file. This config file notably lets you define how to map your input documents to your index fields and whether these fields should be stored and indexed. See the index config documentation.

Let's create an index configured to receive Stackoverflow posts (questions and answers).

# First, download the stackoverflow dataset config from Quickwit repository.
curl -o stackoverflow-index-config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/stackoverflow/index-config.yaml

The index config defines three fields: title, body and creationDate. title and body are indexed and tokenized, and they are also used as default search fields, which means they will be used for search if you do not target a specific field in your query. creationDate serves as the timestamp for each record. There are no more explicit field definitions as we can use the default dynamic mode: the undeclared fields will still be indexed, by default fast fields are enabled to enable aggregation queries. and the raw tokenizer is used for text.

And here is the complete config:

# Index config file for stackoverflow dataset.
#
version: 0.7

index_id: stackoverflow

doc_mapping:
  field_mappings:
    - name: title
      type: text
      tokenizer: default
      record: position
      stored: true
    - name: body
      type: text
      tokenizer: default
      record: position
      stored: true
    - name: creationDate
      type: datetime
      fast: true
      input_formats:
        - rfc3339
      fast_precision: seconds
  timestamp_field: creationDate

search_settings:
  default_search_fields: [title, body]

indexing_settings:
  commit_timeout_secs: 30

Now we can create the index with the command:

./quickwit index create --index-config ./stackoverflow-index-config.yaml

Check that a directory ./qwdata/indexes/stackoverflow has been created, Quickwit will write index files here and a metastore.json which contains the index metadata. You're now ready to fill the index.

Continue on to the Quickwit documentation to add your first documents and execute your first search queries.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Glasskube v0.10.0 out now!

Jake Page — Mon, 24 Jun 2024 11:27:08 +0000

Welcome back to another new release blog post 🚀

This is where we cover the newest shipped features, enhancements, bug fixes and cover all of the recent Glasskube news to make sure you are fully up to speed. We have been riding a continous wave off momentum of internal feature developments as well as interest from the wider community which has led to the delivery of Glasskube v0.10.0.

Let’s check out what you can expect to find in this new minor release.

🚨 Alert: Breaking changes on the horizon ⛓️‍💥

Up until now, Glasskube packages could only be installed once per cluster, which sometimes imposed unnecessary restrictions and limited certain use cases.

From v0.10.0 onwards, the author of a package can specify a "scope," which can be either "Cluster" or "Namespaced" (with the default being "Cluster"). Based on the package scope, the Glasskube system creates either a cluster-scoped or namespace-scoped custom resource. The name of the cluster-scoped CRD is ClusterPackage, while the name for the namespace-scoped CRD is Package. This update introduces a breaking change since we previously used the Package CRD name for cluster-scoped resources. However, we decided to implement this change to align with common Kubernetes nomenclature (e.g., Role/ ClusterRole).

Additional features and UI enhancements

To assist in upgrading to v0.10.0 the glasskube purge command was added to help remove the previous installation.
The glasskube repo update command was also added to fetch the latest package manifests from configured Glasskube package repository.

Access to the full changelog here

Upgrading to v0.10.0

For first time installations, please follow the installation guide here.

To upgrade the Glasskube CLI, Install the newest binary files if you are on Linux or Windows machines.

For macOS, run:

brew upgrade glasskube

To upgrade Glasskube’s cluster components follow to upgrading guide here.

🆕 New Packages integrations available now

Quickwit

Quickwit is a cloud-native search engine that emerged with the goal of creating an open-source alternative to expensive monitoring software like Datadog/Splunk. With its robust Elasticsearch-compatible API, Quickwit integrates well with tooling from the OSS ecosystem, such as Grafana, Jaeger, and OpenTelemetry.

Users are successfully deploying Quickwit at scale, with hundreds of nodes and hundreds of terabytes of data ingested daily, all while enjoying significant cost reductions and how thanks to Glasskube to can get up and running in no time. Quickwit excels in handling logs, traces, security data, and append-only datasets, with plans to support metrics soon.

Hatchet

Hatchet is a distributed, fault-tolerant task queue which replaces traditional message brokers and pub/sub systems, built to solve problems like concurrency, fairness, and durability.

Find installation instructions here.

⏭️ Next packages to be supported

Kubeflow

The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable. Their goal is not to recreate other services, but to provide a straightforward way to deploy best-of-breed open-source systems for ML to diverse infrastructures.

Headlamp

Out of the box, Headlamp is a fully functional Kubernetes UI. By leveraging its powerful plugin system, builders can shape Headlamp to fit their bespoke use-cases, products, and environments.

Velero

Under the VMWare umbrella, Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.

📹 Updated demo video

Check out the newest Glasskube demo video delivered by Philip, where you can find the latest project updates up to v0.9.0.

If you haven’t already, head on over to the Glasskube YouTube channel where you can find the growing archive of weekly Community Calls, Release videos and even some short form content.

☁️ Join Glasskube Cloud

We are starting to build our Glasskube cloud offering to include advanced features in security, accessibility, and team collaboration. Stay updated on our progress by signing up here.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

10 Essential Books to Accelerate your Cloud Career

Jake Page — Wed, 19 Jun 2024 13:17:44 +0000

TL;DR 🤓

DevOps and cloud engineers need not have a floor-to-ceiling bookshelf full of books to enhance their skills. A few key books are all you need (at least at the beginning). Below I highlight key titles for both junior and senior DevOps and cloud engineers, focusing on improving productivity, understanding cloud native technologies and cloud infrastructure. Emphasizing practical application and continuous learning is essential for advancing a career in the rapidly evolving cloud industry.

With the vast amount of valuable and entertaining sources of Cloud Engineering information, from podcasts and YouTube channels to even movies, it can sometimes make us question the authenticity of the well-stocked bookshelves we see behind team members on our daily Zoom calls. Do they actually read those books, or are they just for show?

As someone who has always loved books for their content and entertainment value, I've found myself increasingly drawn to them in recent years for the change of mental pace they provide compared to other modern content mediums.

Unlike other technical book recommendation articles, I want to distinguish between four distinct categories of books that I find helpful to keep in mind. Not every book on this list is meant to be read from cover to cover, nor is every book a dense technical textbook that will gather dust on your shelf after a quick leaf-through.

The handful of book recommendations that have made an impact on my own cloud journey fall into the following categories:

Core concepts: 🏗️

Cloud Computing: Concepts, Technology & Architecture
The DevOps Handbook
Cloud Native Infrastructure

Reference books: 🔎

Site Reliability Engineering
AWS Fundamentals

Page turners: 📑

The Phoenix Project
Accelerate

Not cloud specific but still must reads:📍

Wiring the winning organisation
Slow productivity
Designing Data-Intensive Applications

🚨 Disclaimer: Gene Kim is going to feature prominently on this list

Before we begin

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Core concepts 🏗️

These are the types of books that provide beginners with essential core concepts and frameworks, forming a solid foundation that helps future lessons fall right into place.

1. Cloud Computing: Concepts, Technology & Architecture

Synopsis:

Originally published in 2013, this book is respected industry wide and is a comprehensive guide to cloud computing by exploring its history, models, mechanisms, architectures, and security considerations. Combining case studies with technical analysis, the book examines topics ranging from cloud-enabling technologies to cloud service level agreements.

What you will learn:

You will get a comprehensive and vendor-neutral understanding of cloud technologies, making it easier to assess various cloud solutions and providers objectively.
Fundamental cloud computing concepts, models, and mechanisms.
A deeper understanding of cloud characteristics, security threats, and risk management frameworks, crucial for designing secure and reliable cloud solutions.

Quote from the book:

"There is no greater danger to a business than approaching cloud computing adoption with ignorance. The magnitude of a failed adoption effort not only correspondingly impacts IT departments, but can actually regress a business to a point where it finds itself steps behind from where it was prior to the adoption—and, perhaps, even more steps behind competitors that have been successful at achieving their goals in the meantime."

A review that caught my eye:

Get it here.

2. The DevOps Handbook

Synopsis:

By presenting the “Three ways” as the principles for mitigating these challenges and achieving world-class performance, the book illustrates how organizations can adopt DevOps principles and practices to accelerate delivery, improve reliability, and create a more satisfying and productive work environment.

What you will learn:

The Three Ways (Flow, Feedback & Continual Learning and Experimentation).
Focus on Deployment Lead Time.
Conway’s law.

Quote from the book:

"DevOps isn't about automation, just as astronomy isn't about telescopes."

A review that caught my eye:

Get it here.

3. Cloud Native Infrastructure

Synopsis:

The authors explain key concepts like representing infrastructure through code, APIs, managing application life cycles in a cloud-native way, and ensuring security and compliance, but overall the book stresses that cloud native infrastructure is about adapting principles and processes more than specific technologies, impacting application management as much as hardware.

It guides readers on when and why to adopt these practices, outlining how they differ from traditional approaches.

What you will learn:

Cloud native infrastructure prioritizes building infrastructure with software over manual configuration.
Resiliency is paramount in cloud native infrastructure.
Cloud native infrastructure necessitates a shift in mindset and company culture.
Embracing chaos and designing for failure are essential.

Quote from the book:

"The only systems that should never fail are those that keep you alive (e.g., heart implants, and brakes)".

A review that caught my eye:

Get it here.

Reference books 🔎

Keep these books close by, preferably within arm's reach. In truth, the "Core Concepts" books could also fall into this category and vice versa. As you build your personal home library, you'll definitely want hard copies of these books. This way, you can reference them easily, make annotations, earmark important sections, and fully extract the valuable insights they offer.

4. Site Reliability Engineering (SRE)

Synopsis:

Site Reliability Engineering (SRE), a discipline pioneered by Google, offers a distinct approach to managing large-scale software systems by emphasizing the application of software engineering principles to operations. The book explores this approach in detail, providing insights into Google's production environment, SRE principles such as embracing risk and eliminating toil, and practical practices including monitoring, incident response, and capacity planning.

It emphasizes the importance of automation, simplicity in software design, and a blameless postmortem culture for continuous improvement.

What you will learn:

The core principles and practices of Site Reliability Engineering.
Gain insights into Google's production environment and the challenges of running large-scale systems.
How to apply SRE principles to their own organizations, regardless of size or technical expertise.

Quote from the book:

"If a human operator needs to touch your system during normal operations, you have a bug. The definition of normal changes as your systems grow."

A review that caught my eye:

Get it here.

5. AWS Fundamentals

Synopsis:

"AWS Fundamentals" guides readers through the essentials of Amazon Web Services (AWS), emphasizing practical application over certification preparation. It provides a comprehensive overview of core AWS services like EC2, S3, RDS, DynamoDB, Lambda, and more, categorized by function (compute, database and storage, messaging, etc.

The authors offer practical examples, use cases, configuration recommendations, and tips for each service. Additionally, the book introduces Infrastructure as Code (IaC), explaining its importance and demonstrating how to use frameworks like CloudFormation, Serverless, and CDK to provision infrastructure.

What you will learn:

Understanding the core building blocks of AWS (a lot of core cloud concepts are valid for over cloud providers).
Learning how to apply AWS knowledge in real-world scenarios.
Gaining an understanding of Infrastructure as Code (IaC).

Quote from the book:

“Learning AWS doesn’t need to be hard. It is important to focus on the basics and to understand them well. Once this is done all new services or features can be understood really well.”

A review that caught my eye:

Get it here.

Page turners 📑

Apart from teaching valuable professional lessons these books have felt as entertaining as other non technical works of fiction or non-fiction. All three left me with a strong “I need you to read this“ feeling.

6. The Phoenix project

Synopsis:

A novel (yes an actual novel) that follows Bill Palmer, a VP of IT Operations at Parts Unlimited, who is tasked with salvaging the company's failing IT project, code-named "Phoenix." As Bill navigates the challenges of a stressful work environment full of miscommunication, finger-pointing, and a looming audit, he crosses paths with Erik, a board member who introduces him to the principles of DevOps, without calling it that.

Throughout the story, Bill and his team work to implement these principles, striving to streamline their workflow and improve collaboration between Development and IT Operations. The book is special since the conceptual thrust of the book of spreading DevOps practices is delivered in the refreshing form of a fully fledged narrative novel.

What you will learn:

The three ways is also touched upon in this book (similar to the DevOps Handbook).
The importance of identifying and managing constraints to optimize the flow of work.
The importance of collaboration and communication between Development, IT Operations, and the business as a whole.

Quote from the book:

“Being able to take needless work out of the system is more important than being able to put more work into the system.”

A review that caught my eye:

Get it here.

7. Accelerate

Synopsis:

The book presents four years of research exploring the practices that contribute to high-performing technology organizations. The authors sought to identify the capabilities that drive software delivery performance and, in turn, impact organizational performance. Their findings highlight twenty-four key capabilities, categorized as continuous delivery, architecture, product and process, lean management and monitoring, and cultural, that demonstrably improve software delivery and overall business outcomes.

The book emphasizes that these capabilities are measurable and improvable, offering guidance for organizations to assess their current state and embark on a journey of continuous improvement.

What you will learn:

Software delivery performance is a key predictor of organizational performance
A set of capabilities, including technical practices, Lean management, and a generative culture, drive improvements in software delivery performance.
Transformational leadership is crucial in enabling and amplifying the adoption of these capabilities.

Quote from the book:

"The most important characteristic of high-performing teams is that they are never satisfied: they always strive to get better."

A review that caught my eye:

Get it here.

Not cloud specific but still must reads📍

Not specifically cloud related at all but these are book that for different reasons I feel can round out a cloud engineer.

8. Wiring the Winning Organization

I'm a huge fan of reading "leadership" or "management" focused books, even though I'm not in either of those positions. Learning what leadership should care about and the key ingredients needed to achieve excellent results serves as a cheat sheet for any individual contributor wanting to stand out by being impactful. Change doesn’t always have to come from the top, you can push it from wherever you find yourself.

Synopsis:

The book presents a new theory of performance management, emphasizing how leaders can create the conditions for their organizations to achieve exceptional results. The book introduces three key mechanisms for building a "winning organization": slowification (making problem-solving easier), simplification (making problems easier to solve), and amplification (making problems more visible).

Through a combination of theoretical explanations, practical case studies, and real-world examples, the authors demonstrate how these mechanisms can be applied across diverse industries and organizational contexts to achieve superior performance.

What you will learn:

Greatness in any endeavor is achievable through a focus on refining the "social circuitry" of an organization.
Three key mechanisms slowification, simplification, and amplification can be employed to move an organization from a "danger zone" to a "winning zone".
Leaders who embrace these mechanisms can create organizations that achieve extraordinary results.

Quote from the book:

“Slowification enables the shift from reactive, "fast thinking" based on ingrained habits to more effective "slow thinking" that allows for deliberation, reflection, and creativity in problem-solving.“

A review that caught my eye:

Get it here.

9. Slow productivity

Synopsis:

Slow Productivity by Cal Newport challenges the modern obsession with visible busyness as a measure of productivity, what he terms pseudo-productivity. Instead, the book champions a slow productivity philosophy based on three core principles: doing fewer things, working at a natural pace, and obsessing over quality.
This philosophy argues that by intentionally limiting workloads, embracing a sustainable work pace, and prioritizing quality over quantity, knowledge workers can achieve greater meaning and produce superior results.

The book explores the theoretical underpinnings of these principles and offers practical strategies for implementing them in various professional settings.

What you will learn:

To do less
Work at the more natural pace
Obsess over quality

Quote from the book:

"for all of our complaining about the term, knowledge workers have no agreed-upon definition of what “productivity” even means."

A review that caught my eye:

Get it here.

10. Designing Data-Intensive Applications

As cloud engineers, we collaborate closely with the developers in our organization who write the applications we deploy and maintain. While this book doesn’t provide directly actionable knowledge for our day-to-day tasks, it has given me a deeper understanding of the concepts underpinning the creation of our organization’s apps. More importantly, it has provided me with a common language to effectively communicate with the developers on my team.

Synopsis:

Designing Data-Intensive Applications, by Martin Kleppmann, explores the fundamental principles and practical considerations for building reliable, scalable, and maintainable data systems. The book examines various data models, including relational, document, and graph-based models, and their respective query languages, analyzing their suitability for different applications.

It also examines storage engines, data encoding formats, and schema evolution. A key focus is the exploration of distributed data systems, including replication, partitioning (sharding), and the challenges of maintaining consistency and consensus in such environments. The book uses real-world examples of successful data systems to illustrate key concepts and trade-offs.

What you will learn:

Various data models, including relational, document, and graph-based models, and their respective query languages, analyzing their suitability for different applications.
Storage engines and how databases arrange data on disk to find it again efficiently
Reliability, scalability, and maintainability.

Quote from the book:

"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair."

A review that caught my eye:

Get it here.

If you like our content and want to support us on this mission, we'd appreciate it if you could give us a star ⭐️ on GitHub.

⭐️ Star us on GitHub 🙏

Glasskube reaches 1k stars 🌟

Jake Page — Mon, 10 Jun 2024 09:04:10 +0000

I was trying to thinking of sayings or mantra’s ambitious people repeat to themselves to keep themselves motivated. A few came to mind:

"Fortune favors the bold."
"No guts, no glory."
"You miss 100% of the shots you don't take."
"Nothing ventured, nothing gained."

Many of us have heard these sayings or something similar before, what you have less often which I think is equally important maybe doesn’t fit that well into a motivating one liner but might go something like this:

Sure, shoot for the stars but make sure to celebrate every win along the way. Because the journey is equally as important as the eventual goal.

While nobody starts a project with the sole aim of getting 1,000 GitHub stars, the entire Glasskube team is incredibly proud of reaching this milestone so quickly. I wanted to take a moment to reflect on our rapid organic growth and celebrate our achievements.

What is Glasskube?

Glasskube is the next-generation Kubernetes package manager, now available in its beta version. With Glasskube, you can effortlessly install, upgrade, configure, and manage your Kubernetes cluster packages. Check out the available packages here. Glasskube streamlines repetitive and cumbersome maintenance tasks, making cluster management a breeze. The project is evolving rapidly, with new functionalities being shipped with every weekly release.

So go ahead, dive in, explore, and share your thoughts with us! Your feedback is incredibly valuable as we strive to make Glasskube the best Kubernetes package manager out there. We're also beginning to develop Glasskube Cloud, building on the requests from our current users to enhance the tool further. We're eager to hear which features you'd like to see included in the Glasskube Cloud offering.

The best way to stay in the loop is to sign up for Glasskube Cloud and be informed when it's ready: https://glasskube.cloud/.

The story of our organic growth so far

Having oficially launched the project in February 2024 we are super happy to have hit the 1000k milestone in the first days of June.

Let’s take a look at some of the work be have been doing to make this happen.

We’ve been shipping 🚢

In the past few months, our team and a growing legion of open-source contributors have helped us ship eight minor version releases, the latest being v0.8.0. You can find the full changelog here. Jakob and Christophe, both Glasskube developers, have been hard at work rolling out new features and functionalities. They've also been invaluable in providing feedback and managing the PR review process, ensuring that code is merged swiftly and safely.

Every contributor, found on the Glasskube Discord server, has brought something valuable to the community. However, a few deserve a special shoutout: Hanshal, Utkarsh, and Baalakshan. These contributors have been diligently submitting valuable PRs, addressing open issues, and advocating for Glasskube wherever they go.

We've been organizing weekly community calls every Monday on the Discord server. These calls have been a fantastic opportunity to share news, align expectations, and grow together. Some contributors have even given short talks exploring concepts like Kubernetes CRDs, presented by Hanshal, and Chaos Engineering, shared by Baalakshan.

Meetups

Cofounders Philip and Louis, myself and Community members have been really active attending meetups and conferences around Europe and India. We were at a few KCD’s, AWS Summit Madrid and other CNCF meetups too. It has been a great way to spark conversations and share the work that’s being done.

Partnerships 🫂

We have partnered with third party packages such as Keptn and Quickwit to expedite their integration onto Glasskube.

You can now find installation steps right in the documentation of the supported packages.

Content creation 📚

As we build and continually refine a quality product to support our ever-expanding user base, we aim to be a consistent source of valuable content. This includes in-depth Kubernetes topics like this, more technical product explorations like this, and even broader opinion pieces like our Git guide and our article on the current state of the DevOps scene.

Check out our latest launch video 🚀

Building Glasskube so far has been such a gratifiying experience, on the one hand we are connecting and understanding the issues so many cloud practitioners are having in there efforts to deal with Kubernetes Package management in their daily routines. Better understanding them and delivering on their requests is special.

On the other hand, collaborating with so many stellar open-source community contributors is a massive perk and added bonus of building an OSS tool in public. Much of our early growth can be directly attributed to these outstanding OSS contributors. We hope to continue this collaboration as we aim for 2, 3, and 10k stars, and more importantly, to make Glasskube a tool installed in Kubernetes clusters far and wide.

If you get value from the work we do, we'd appreciate it if you could
⭐️ Star Glasskube on GitHub 🙏

DEV Community: Jake Page

Four Layers of Validation in Kubernetes with Claude Code

Layer 1: Skills (passive, shaping what gets generated)

Before and after

What skills can’t do

Layer 2: Commands (active, checking what was generated)

What commands can’t do

Layer 3: Integration tests (runtime, proving it works)

What this catches that the other layers can’t

Layer 4: Human review in a real environment

Putting it together: agents handle layers 1–3, humans handle layer 4

Adopting the layers

Skin In The Game

Glasskube Launch week #2: The wrap up

TL;DR

A feature a day 📅

Successful Show Hacker news launch 📰

Top 3 dev hunt 🚀

Our first every San Francisco meetup 🌉

Community feedback 🗣️

Reaching key metrics 📊

2024: A new frontier for Kubernetes package management

Introduction

Why Kubernetes needs better package management

What is Glasskube?

Current tools: Helm and Kustomize

Problems with existing tools

Care to support us?

Glasskube explained

How Glasskube works

Main features

Solving common issues

How to set up Glasskube

What you need to install

Step-by-step installation guide

Using Glasskube: Key functions

Access the GUI

Install packages

Managing package dependencies

Handling package configurations

Working with GitOps

Automatic updates with Renovate

Adding custom package repositories

Glasskube vs. other package managers

How it differs from Helm and Kustomize

Business service offerings

What's next for Glasskube

New features coming soon

Wrap-up

The GitOps Kubernetes starter template that gets you set-up in minutes instead of hours

The next logical extension for infrastructure IaC

What is a Glasskube package?

Use ArgoCD to deploy Glasskube packages

Template structure

Cluster setup

Prerequisites

Install the Glasskube CLI

How to use the template

The result

Managing your cluster

Using the CLI

Through the UI

Installing packages

Updating packages

Manually

Automatically with Renovate

Uninstalling packages

Updating Glasskube

Working with Apps

Monitoring with kube-prometheus-stack

Setting up a postgres dashboard

Template setup walkthrough

Upcoming Features

Private Repo Support

Improved Renovate Integration

Improved Dependency Resolution

Summary

Feedback

Why building an IDP for Series A companies doesn't make sense

Setting the right foundations

Monitoring with `kube-prometheus-stack`

Prerequisites