DEV Community: Innovation Process Technology AG (ipt)

Generative UI - The Future of Human-AI Interaction

Weili Gao — Thu, 28 May 2026 05:00:00 +0000

This article was co-authored with my colleague

Benjamin BürgisserFollow

Hi, I'm Benjamin, a software engineer and IT architect based near Zurich, Switzerland. https://b-dimension.com/

TL;DR

Generative UI is the next evolution of AI interaction, moving beyond text to create dynamic, agent-driven user interfaces. We explored three key patterns - Static (AG-UI), Declarative (A2UI), and Open-ended UI - and used the CopilotKit framework to build two demos: a Tic-Tac-Toe assistant, and a 2D RPG with LangGraph NPC.

Why Generative UI?

Today’s AI agents primarily communicate through text. While revolutionary, this often forces humans to interact with a conversational interface, even for tasks that would be more efficient or intuitive within a traditional graphical user interface.

Generative UI is the paradigm shift that enables AI agents to dynamically generate the UI elements required to complete a task. Instead of the user describing what they see, the agent reads the UI state directly and acts through it - clicking buttons, moving characters, proposing decisions.

This shift blends the power of LLMs with familiar interactive components and is critical for implementing human-in-the-loop workflows, where the agent can propose an action but requires human confirmation via an interactive UI before execution. This ensures control and a better user experience.

The Three Core Patterns of Generative UI

Three core patterns exist for building Generative UI:

Static Generative UI (AG-UI): the agent generates a command (like a function call) for a predefined frontend tool. The UI for that tool is already built and the agent simply triggers it.
Declarative Generative UI (A2UI, Open-JSON-UI): the agent generates a structured data format (like JSON) that describes the UI it wants to render (e.g., a form, a chart, or a list). The frontend then interprets this data and renders the elements.
Open-ended Generative UI (MCP Apps): the agent generates raw, open-ended content, such as a block of custom HTML/CSS, which is then rendered directly in the UI. This is the most flexible approach but requires careful security measures.

The different patterns are not mutually exclusive and can be complementary, e.g. both AG-UI and A2UI are used by the CopilotKit framework for implementing Generative UI.

Key Generative UI Frameworks

The Generative UI approach is seeing rapid development from several frameworks:

CopilotKit: a comprehensive framework for building agent-driven interfaces.
Pillar: a simpler, headless framework focused on the core logic of agent tool routing.
Vercel AI SDK: provides foundational support for streaming and model integrations, and is used as the runtime for CopilotKit's built-in agent.

Introducing CopilotKit

We picked CopilotKit for building our two demos. It is an open-source (MIT) framework with 30k+ stars on GitHub that connects AI agents to your frontend. It is the company behind the AG-UI protocol.

Its core concepts are:

Readables: expose app state to the agent (board positions, inventories, distances).
Actions: let the agent do things in your UI (make a move, take an item, spawn a reward).
Chat UI: a themed sidebar or popup, with streaming and tool-call rendering, out of the box.

It supports React/Next.js and Angular, and works with agent backends like LangGraph or Microsoft Agent Framework. CopilotKit is partnered with Microsoft and Google for development on the AG-UI and A2UI protocols.

Our Demos

To gain hands-on experience with Generative UI, we have developed two demos with CopilotKit: a Tic-Tac-Toe Assistant and a 2D RPG. They demonstrate how a front-end application can expose its state and functionality (tools) to an agent, allowing the agent to perform actions that are visible and interactive for the user.

Demo 1: Tic-Tac-Toe Assistant

In our first demo we built a tic-tac-toe assistant that coaches the player at the immensely sophisticated game of tic-tac-toe. The agent can suggest moves (with an accept/reject dialog for human-in-the-loop), annotate the board with threats and opportunities, and render a custom "Coach Card" via generated HTML. The demo illustrates key Generative UI features by applying the patterns discussed above:

AI Move Suggestion (AG-UI + Human-in-the-Loop): The agent calls a predefined proposeNextMove tool, and the result is rendered as an "Accept / Reject" dialog (renderAndWaitForResponse), requiring user confirmation before the move is made.

Board Annotations (AG-UI): The agent calls an analyzeBoard tool, which returns data used to render overlay symbols (threats, opportunities) directly onto the existing UI components (Board.tsx/Square.tsx).

Coach Card (Open-ended Generative UI): The agent calls a renderCoachCard tool, which generates fully custom HTML for the "Coach Card" component, rendered using a secure mechanism (dangerouslySetInnerHTML with DOMPurify).

Shared Game State (Agent Context / Readable State): The useAgentContext hook is used to publish the current board state, player, and winner status, giving the agent a "readable state" of the application.

The application is structured as a Next.js app, with all frontend logic (tools, UI) in the browser, communicating with the backend API route (/api/copilotkit) which hosts the CopilotKit’s built-in agent and handles the model interaction (Azure OpenAI gpt-5-mini via @ai-sdk/azure).

This demo validated that exposing React state as readables and defining actions as tools is a clean, working pattern. The built-in agent made setup fast - readables and the instructions prop on the chat sidebar just worked.

Demo 2: 2D RPG with LangGraph NPC

For the second demo we wanted to go deeper: custom system prompts, richer agent logic, and a more complex scenario. We built a 2D RPG where the player walks around a medieval world collecting bananas, berries, and crystals. An NPC - powered by a LangGraph agent - acts as a quest giver. The agent can:

See the game state: positions, inventories, objects on the map, player–NPC distance.
Move the NPC through the world using A* pathfinding around obstacles.
Give quests dynamically based on what's available on the map. Take items from the player and reward coins for completed quests.

All of this runs through CopilotKit's readables and actions - the agent never touches the DOM directly; it just reads structured state and calls named functions.

A Quest in Three Acts

Nothing is scripted. The NPC reads the world state, decides what to ask for, and reacts to whatever the player actually does.

Act 1 - The Quest: We ask the NPC for a quest. It inspects the map and decides: "Bring me 3 crystals." No quest table - just the agent reasoning over game state.

Act 2 - The Pivot: Instead of crystals, we grab a banana and call the NPC over. It pathfinds around the brick walls to reach us. In a scripted game, offering the wrong item would be a dead end. Here, the agent goes off-script and reluctantly accepts the banana anyway.

Act 3 - The Reward: We ask for a reward. The agent decides a single coin is fair, spawns it next to the NPC, and we pick it up.

The NPC feels more real because we can freely talk to it and it makes decisions on the fly - exactly the kind of dynamic interaction that's hard to achieve with traditional game scripting.

Lessons Learned

Summary of what we've learned from the demos.

What Worked Well

The readable/action model is clean. You declare what the agent can see (useCopilotReadable) and do (useCopilotAction) right next to the React components that own the state. CopilotKit handles serialization, transport, and tool binding. It feels like a natural extension of React's component model - state flows down, actions flow up, and the agent plugs into that loop.
The chat UI saved real time. A themed sidebar with message history, streaming, and tool-call rendering - for free. We styled it to match our medieval parchment theme with a few CSS custom properties.
LangGraph integration is straightforward. Define a graph, export it, point the Next.js route at the deployment URL. CopilotKit's state annotation passes readables and actions through cleanly.

What Tripped Us Up

Readables don't auto-inject with custom agents. This was the biggest surprise. With the built-in agent (as in the Tic-Tac-Toe demo), useCopilotReadable values appear in the model's context automatically. With a custom LangGraph agent, they arrive in state.copilotkit.context - but you have to manually inject them into your system message. Actions worked immediately; readables silently did nothing until we traced the issue.
The instructions prop is ignored with custom agents. We set system instructions via CopilotSidebar's instructions field and they were silently dropped. This is a known open CopilotKit issue. The workaround: define instructions in the agent's system message directly.
React's async batching vs. rapid tool calls - when the agent fired two rapid calls (e.g. "take 1 berry" then "take 3 crystals"), the second read stale state because React hasn't re-rendered yet. We had to move from useEffect-based ref syncing to immediate ref updates - a useful pattern when bridging async agent actions with React state.

Is Using CopilotKit Worth It?

Couldn't we just append state to the user's message and define function-calling tools on our own? Yes - but CopilotKit gives you:

A structured protocol - AG-UI standardizes the agent–frontend connection so you don't reinvent the wheel.
A full chat framework - sidebar, streaming, theming. Non-trivial to build and maintain yourself.
A component-level API - readables and actions co-located with the components that own the state.

For a quick prototype, rolling your own is fine. For anything that will grow - multiple agents, generative UI patterns, shared state - a framework pays off.

Conclusion

Generative UI represents an exciting step forward in AI-powered applications, moving beyond simple text generation to dynamic, context-aware user interaction. Our experience with CopilotKit in developing the two demos validated the effectiveness of the AG-UI and open-ended patterns for creating engaging and powerful Human-in-the-Loop experiences. As frameworks mature, Generative UI is poised to become a core part of how developers integrate AI into modern web applications.

From Rot to Rhythm: How We Rebuilt Our Documentation Culture with Docusaurus

Vincent von Büren — Tue, 19 May 2026 06:21:49 +0000

The Forgotten Layer

Every infrastructure has a critical layer that doesn't show up on dashboards, but without it, everything slows down: documentation.

We learned this the hard way.

A little background for you. My team and I provide cloud services on a private cloud. This means that our customers are developers. We deliver platform features, portal functionality, maintenance, and infrastructure services. Over the last years, we invested heavily in automation and feature delivery — the things every PO dreams of. That focus allowed us to onboard more and more platforms onto our cloud service offering.

But there was one thing we consistently deprioritized: documentation.
And our customer-facing service docs paid the price.

For years, it lived in Confluence spaces — large, tangled forests of pages where knowledge went to die. And of course, search functionality is also particularly bad using Confluence. So whenever a client reached out and asked if this part of the service information was still valid, the advice was always the same:

"I am not sure, let me double check."

And that worked — until it didn't.

This post is not just a reflection. At the end, I will also share a Docusaurus scaffold we built internally — including linting, theming, and contribution workflows — so you can replicate this approach without reinventing it.

2. The Slow Decay

Documentation debt doesn't scream. It whispers.

At first, it was a broken link here, an outdated diagram there. Nothing dramatic. But as our platform grew, the gap between what we shipped and what we documented widened.

Onboarding stretched from a week to months.
Customers filed support tickets for information that should have been one search away.

When we looked closer, the symptoms all pointed to the same root cause:

Docs lived outside the development flow
Ownership was unclear
Updating them felt like overhead
And sorry for the dinosaurs out there: Confluence!

3. The Moment Everything Snapped

The real tipping point came in the form of a short email from a customer.

“This page hasn't been updated since 2022.”

We checked. They were right.

Behind that single outdated page lay dozens more in similar shape. We could have patched the page and moved on. But deep down, we knew:
this wasn't a page problem — it was a structural problem.

That was the day we stopped treating documentation as a side activity. We started treating it as part of the product.

4. Why Confluence Couldn't Keep Up

Confluence had served us well in the early years. But we had outgrown it.

Here's why it broke down at scale:

Disconnected from development: Docs lived in a separate place, updated weeks after features shipped (if at all).
DevEx: Adding interactive code snippets, or embedding multi-media content is all possible in Confluence, but the writing is cumbersome. Markdown or AsciDocs for the win.
No versioning: We couldn't align docs with specific releases.
Search pain: Finding anything meant traversing endless page trees.
No customer contribution: Feedback loops were slow, often going nowhere.
No guardrails: No linting, no validation, no CI — which meant quality drift.

And perhaps most importantly:
Confluence didn't create shared ownership.

It was "someone else's job."

5. Before the Tool Came the Story

This might sound counterintuitive for an engineering team, but we didn't start with a tool.

We started with a story.

"If documentation isn't close to the code, it will always be outdated."

This became our North Star. We repeated it in every retro, every cross-team architecture session, and every leadership conversation.

We didn't want just better docs. We wanted documentation to become part of the way we build — fast, iterative, owned by everyone, and open to customers.

Once that vision was clear, the tooling decision became much easier.

6. Choosing the Right Documentation Platform

We ran a structured evaluation of several tools. We needed something that could:

Git-native workflows
Versioning
Fast performance for customers
Low barrier to contribution
Enough flexibility without becoming a framework project

Tool	Pros	Cons	Verdict
Confluence	Familiar, structured	Poor dev integration, outdated model	❌ Outgrown
Read the Docs	Simple, Sphinx-based, versioning support	Python-focused, limited customization	⚠ Too rigid
MkDocs	Lightweight, good search, fast	Less UI flexibility	⚠ Great but limited for branding
Hugo	Very fast, flexible	Requires more setup, not docs-first	⚠ Too generic
Docusaurus	MDX, versioning, React components, fast CI/CD integration	Initial setup heavier	✅ Perfect balance

We chose Docusaurus because it aligned with how we already worked: Git, Markdown, pull requests, CI/CD, and reusable components.

7. Understanding Docusaurus (for those new to it)

At its core, Docusaurus is a static site generator designed specifically for documentation.

It combines Markdown with React — giving you the simplicity of writing content in .md files and the flexibility of building custom UI when needed.

Key features that mattered for us:

Versioning: Tie docs to releases — a feature Confluence never had.
MDX support: Mix Markdown with React components for rich documentation (feels funky, but kind of cool).
Search: Fast, extensible, works out of the box (Algolia or Typesense).
Custom plugins: Integrate linters, search, CI checks.
Speed: Our site loads fast — customers notice that.
Familiar workflow: Pull requests, code review, previews — like any other repo.

For our engineers, writing docs started feeling less like "extra work" and more like writing code. Some even said that documentation started to be fun.

8. Our Docusaurus Documentation Platform

Instead of treating documentation as a collection of pages, we built a documentation platform on top of Docusaurus.

The goal was simple: any team should be able to start migrating from Confluence to Docs-as-Code without needing hand-holding.

Our Docusaurus came with structure, guidance, and guardrails that made contribution predictable and safe.

What our Docusaurus setup provides

Our documentation platform includes:

Clear structure and navigation: A predefined information architecture so teams don’t invent their own page trees.
Built-in tutorials: Step-by-step guides that explain how to migrate content from Confluence, how to structure pages, editing guidelines, and how to use MDX components correctly.
Guardrails by default: ESLint & Prettier for consistent formatting
Vale for terminology, style, and writing quality
CI checks that prevent broken builds or low-quality content from being merged
Reusable MDX components: Components like , , and warnings for deprecations helped teams present information consistently without custom markup.
Search and discoverability: Fast, global search made content immediately usable for customers.
CI/CD with preview environments: Every pull request gets a preview deployment so contributors can see exactly what they are about to publish.

Theming and branding: Documentation looks and feels like part of the platform, not an afterthought.

The result: teams could migrate their content incrementally, at their own pace, without breaking anything — and without needing a central documentation team to mediate every change.

Sharing is Caring!

Later in this post, we'll share a generic version of this setup — a reusable Docusaurus scaffold based on what we built internally. It includes the same structure, guardrails, and conventions, so your team can adopt Docs-as-Code without starting from scratch

9. The First Migration

We started small: one service, one team.

We rewrote only the most accessed and most broken pages. Minimal navigation. Strict linting. Then we shipped.

Then we shipped it.

The response surprised us.

Customers noticed the speed
Engineers fixed docs in the same PR as code
Reviews made documentation visible again

Momentum follows. What used to live across multiple Confluence spaces consolidated into a single, structured documentation system.

10. Documentation as an Enabled Ecosystem

The biggest shift wasn't technical — it was systemic.

Documentation became a shared interface between platform teams, SREs, portal developers, and customers.

It enabled self-service instead of support load
It reduced onboarding friction
It made platform capabilities — and gaps — visible and discoverable
It allowed feedback to flow directly into improvements

In other words: documentation stopped being static knowledge and became part of the platform ecosystem itself.

In hindsight, this aligns closely with what platform engineering maturity models describe as an enabled ecosystem: teams don't just consume the platform — they actively shape it.

11. What Surprised Us Most

When we moved to Docs as Code, we expected better structure.
What we didn't expect was how it would bring teams together.

SRE engineers and portal developers now speak the same documentation language. Platform teams see their work reflected directly in docs. Customers trust what they read again.

The act of writing and reviewing together created something Confluence never could: a shared documentation identity.

12. Lessons Learned Along the Way

Start with a narrative, not a tool.
Align teams on why you're doing this.
Don't migrate everything.
Migrate the docs that matter most first. Cut the old and ugly and don't be afraid to rewrite stuff
Make contribution effortless.
A good scaffold lowers the barrier.
Automate quality.
Vale, CI, formatting — make good docs the default, not the exception.
Documentation is never done.
Build habits, not one-time migrations.
Customers are collaborators.
Give them a way to contribute — it builds trust.

13. Why We're Sharing Our Scaffold

This isn't just a success story. It's also a pattern other teams can use.

We’re making our internal scaffold available as a private starter repo: https://github.com/VincentvonBueren/erfa-docusaurus-demo/

You can fork it, clone it, or just learn from how we structured it.

We don't believe great documentation should be an afterthought.
We believe it should be a first-class citizen of every cloud platform.

14. The Moral of the Story

Our journey began with a customer pointing out an outdated page.
It ended with documentation that moves at the speed of our platform.

Docusaurus wasn't the hero of this story.
It was the vehicle that helped us tell a new one.

If your team is still fighting Confluence sprawl, don't start with a tool.

Start with a narrative. Then give it the right structure. And watch your docs come alive.

Beyond the Pod: Why wasmCloud and WebAssembly Might Be the Next Evolution of the Platform

Jakob Beckmann — Tue, 21 Oct 2025 05:08:44 +0000

Over the past few months I have invested some time to contribute to an open source project I find fascinating: wasmCloud. As a platform engineer and architect, I am very familiar with how software platforms are typically built in practice. However, with the ubiquity of Kubernetes, you run the risk to being stuck in the "doing it the Kubernetes way" line of thinking. But then again, are there any better ways? This is where wasmCloud caught my attention. A modern platform building on proven concepts from Kubernetes, but with some significant differences. In this article I want to introduce wasmCloud, how it compares to Kubernetes, what its internal architecture looks like, and what ideas are, in my humble opinion, a step up from "the Kubernetes way of things".

Before getting started, I need to get some things out of the way. This article will make quite a few comparisons to Kubernetes and bytecode interpreters like the JVM. If you are unfamiliar with these technologies, it might make sense to have a short look at what these are. Considering you clicked on this article, I am however guessing that you are familiar with them and have some experience in platform engineering practices, either as a poweruser of a platform, or as a designer and developer of one.

Moreover, I want to thank the company I work for, ipt, for allowing me to invest time to learn about new technologies such as wasmCloud. Not only is contributing to open source a great way to pay back a community powering the modern world, it is also a huge passion of mine. Being able to help the development of such projects during paid worktime enables me to learn so much on emerging technologies, and maybe help build the revolutionary tools of tomorrow.

So... wasmCloud!? I have been interested in WebAssembly ever since it promised to replace JavaScript, a language I personally consider as extremely poorly designed (someone once told me it was designed in three days, so no wonder there). While WebAssembly is very far from doing anything close to replacing JavaScript in the browser, it has evolved into something else: an application runtime and a potential replacement for containers.

WebAssembly as a Platform Foundation

Modern platforms nearly all build on top of containers as their foundational element to run executable code. This is a logical evolution from Docker's meteoric growth, and the ecosystem that grew around its open standards (such as the OCI - Open Container Initiative). While containers provide a huge step in terms of ease of use, standardization, and security compared to shipping raw artefacts to virtual machines, as was the case before them, they do have some shortcomings.

First and foremost, containers are not composable. In part due to their flexibility, they do not offer standard ways of expressing how the world should interact with them at runtime, or what they rely on to perform their functionality. This means that containers are typically deployed as REST-based microservices, where containers communicate with one another over a network using APIs agreed upon outside of the container standards. This lack of standardization makes building reusable components more challenging than it has to be. Moreover, each container essentially needs a server, authentication, authorization, and more to run. This results in quite some waste in the compute density of the platform, with lots of compute wasted on boilerplate.

Moreover, while containers are a huge step in the right direction in terms of security, they are not quite as secure as most people are led to believe. Containers are "allow by default" constructs, which take quite some work to properly harden.

Finally, due to how containers are typically built, their startup times are not that great. It is not abnormal to see container start times in the dozens of seconds. This does not bother people very much because containers are mostly used to run long running processes (since we need these REST APIs everywhere). However, a large part of containers are mostly idle, waiting for some API request to come in. If one considers that workloads could be called (and thus the process started) only when needed, startup times over 100ms is considered slow.

This is where WebAssembly comes it. WebAssembly addresses these challenges. Composability is addressed by the component model.

WebAssembly: The Component Model

The component model is a way that WebAssembly modules can be built with metadata attached to them which describe their imports and exports based on a rich type system. Moreover, they are composable such that a new component can be built from existing components as long as the imports of one are satisfied by the exports of another. This means that components can interact with one another via direct method/function calls, whose specification is fully standardized. This interface specification is declared in a language known as the WebAssembly Interface Types (WIT) language. An example of a WIT specification of a component relying on a system clock can be seen below:

package wasi-example:clocks;

world mycomponent {
    import wall-clock;
}

interface wall-clock {
    record datetime {
        seconds: u64,
        nanoseconds: u32,
    }

    now: func() -> datetime;

    resolution: func() -> datetime;
}

WIT can be compared to the Interface Definition Language (IDL) from gRPC but for wasm components.

This declaration says that the component relies on an interface wall-clock (it imports the interface) which defines two functions: now and resolution. Both take no arguments and return a datetime object consisting of a seconds and nanoseconds field. This component could then be composed with any other component which exports this wall-clock interface.

If this were a container which would rely on accessing some API, we would need to read a non-standardized documentation of the container image, and then read up on other containers to ensure they provide APIs that match the ones called by the first container.

The WebAssembly component model can essentially be seen as a form of contract-based programming to formalize interfaces between WebAssembly core modules.

WebAssembly: Secure by Default

Whereas containers provide some form of security by namespacing processes and filesystems, WebAssembly actually sandboxes modules such that they cannot affect one another, or the host they run on. By default a WebAssembly module cannot perform any privileged action and needs to be granted explicit permission. I will not dive deeper into the details of this or I might loose myself in a rant on how software security in the modern day and age is abysmal.

WebAssembly: Performance

WebAssembly's main goal is performance. This means that WebAssembly modules run fast, but also that loading modules and starting them is much faster than containers. This has proven to be very useful already, for instance in use cases such as serverless computing, where hyperscalers heavily rely on WebAssembly as a runtime to reduce cold start times, and reduce the delay in function calls.

Considering the idea to avoid having long running servers providing REST APIs and move to raw function calls on short running modules, having extremely short start times is imperative.

Alright, so we can see that WebAssembly can be a great choice for the foundation runtime of a platform. So where are platforms leveraging this? Well, actually, quite some "platforms" leverage this idea already. For instance, SpinKube does exactly this, enabling to run WebAssembly functions on Kubernetes. However, you still interact with these functions via a REST call. Another example is Kubewarden, leveraging WebAssembly modules to evaluate policies. While some might argue that this is not a platform, Kubewarden provides a runtime for arbitrary programs, including their scheduling and deployment. Sounds like a platform to me.

Finally: wasmCloud! wasmCloud is probably what people would consider the closest to a full blown platform to run WebAssembly modules. In other words, what Kubernetes is to containers, wasmCloud is to WebAssembly components. It provides a way to deploy, schedule, link, and lifecycle WebAssembly components on a distributed platform.

wasmCloud Architecture

Let us look at the wasmCloud architecture a little.

This section will contain quite a few comparisons to Kubernetes concepts.

Generally, the wasmCloud architecture can be seen as quite similar to the Kubernetes architecture, with the difference being that wasmCloud does not provide as much flexibility in swapping out building blocks as Kubernetes does. This makes sense as it is a more nascent technology and is currently more opinionated.

As a reference, here is the diagram wasmCloud uses to provide an overview of the platform:

As one can see, the architecture is essentially a set of hosts connected via a so called "lattice". Thus, the architecture distributes the runtime over a set of compute instances in order to achieve resilience against hardware/compute failures. The principle is identical to the one from Kubernetes, providing a cluster in order to be able to quickly shift payloads on the platform to different nodes in case of node failures.

Hosts

wasmCloud hosts are the foundation of the compute platform that is provided. They are the equivalent of Kubernetes nodes and provide a WebAssembly runtime for components to run on. Just as with Kubernetes nodes, application developers will rarely need to worry about the hosts other than for deployment affinities and the like.

In practice, hosts can be anything from a virtual machine, an IoT device, or even a pod running on Kubernetes. In fact, hosting wasmCloud on Kubernetes is a relatively straight forward way to get started with the technology, providing wasmCloud as an application runtime, while providing services via Kubernetes.

Lattice

The wasmCloud lattice is its networking layer. This can seem a bit strange when considering that this a NATS instance.

For those unfamiliar with NATS: it is an event streaming component similar to Kafka, but provides additional features such as a key values store, an object store, and publish-subscribe capabilities.

Having a NATS instance as the "networking layer" confused me quite a lot at first. However, one has to remember that thanks to the component model, we no longer require HTTP/TCP network calls for our components to interact with one another. Thus we don't necessarily need an IP to address a component we want to reach. Of course NATS itself will require a physical network to run on in order to distribute events to its different instances, but wasmCloud then only needs to use NATS.

Essentially, every component exposing a function becomes a subscriber to a queue for this function on NATS. Other components can then call this function via wRPC (gRPC for WebAssembly) by publishing a call to some subject. This is quite different from Kubernetes networking, where calls need to know the location of the callee in the network. Using a subject-based addressing model simplifies deployment and improves scaling and resilience.

As a user of wasmCloud, you do not need to worry about this though. How function calls are preformed under the hood is abstracted away from the user.

This distributed networking aspect is one of the superpowers of wasmCloud, as one does not need to worry about how to address a component on the platform. However, it can also introduce strange behaviour in some cases. For instance, on Kubernetes, it's common sense that a HTTP call to a different pod running on the cluster can fail. On wasmCloud however, if the interface we are calling from a different component returns some type, we use the component like a raw function call in our components code. What if that call fails, not because of the called component but due to a networking issue? In the current implementation of wasmCloud this will lead to a panic in the caller. As this is typically not the desired outcome, efforts are underway to design an adapted way how the interfaces need to be designed to handle failures in the transport layer. On top of that, function calls might change such that might avoid using NATS as a transport layer if the component being called in on the same host and the caller.

Capabilities

This is where Kubernetes and wasmCloud start differing in their philosophy. Thanks to the standardized way interfaces can be declared in the component model, one can describe an abstract interface which provides some functionality, without providing an implementation. This is what capabilities are. They are abstract interfaces that describe some useful functionality, such as reading and writing to a key value store, or retrieving some sensitive information from a secured environment. These capabilities are published on wasmCloud for applications to use.

An application developer can then write a component that makes use of that interface if he/she needs that functionality. He/she does not need to worry about how this capability is implemented. He relies on the "contract" provided by the capability.

In my opinion, while this is quite challenging to grasp initially, this is what makes wasmCloud so promising. Having worked on many platforms in the past, the main challenge is always how additional services can be provided on top of raw platforms such as Kubernetes in a way that makes then highly standardized while easily consumable. In the current state of platform engineering, this quickly becomes a question of good product management. Unfortunately, doing this correctly is surprisingly difficult. Capabilities provide a technical solution to this, with the only limitation being complete incompatibility with existing software.

Providers

A provider is a specific implementation of a capability. For instance, taking the example of the capability enabling the reading and writing to a key value store, a provider might implement this by having a ValKey instance backing the capability. Another provider might implement the very same capability using NATS, Redis, or even an in-memory key-value store.

Abstracting the provider away from the consumer via a capability enables the platform to swap providers based on needs. Of course performing such a swap might be quite complex, for instance involving a data migration from NATS to ValKey. However, the beauty is that the applications do not require any changes as would be the case in traditional platforms.

It should be noted that the provider might run completely outside of wasmCloud itself. However, wasmCloud also provides internal providers that are backed into the hosts themselves, providing functionality such as logging or randomness.

Components

Components refer to the WebAssembly payload that contain your business logic. In the traditional sense, this is your application. However, in wasmCloud lingo, an application is a set of interlinked components including all information about what capabilities they require.

Applications

Applications are an abstraction enabling to declaratively define a combination of components, capabilities, and providers together into a deployable unit. Applications are based on the open application model (OAM) and should thus look quite familiar to people working with Kubernetes. In terms of definition, they are similar to a Kubernetes Deployment, describing not only the deployment unit (component or pod in the Kubernetes context), but also its replication, affinities, links to capabilities, etc.

It should be noted that in wasmCloud v2, applications are re-worked to be much more closely modelled after Kubernetes Deployments and ReplicaSets. Version 2 drops the idea of Applications alltogether and uses Workload, WorkloadReplicaSets, and WorkloadDeployments objects. These are also no longer linked to the OAM. In all likelihood we will write another blog post showcasing the capabilities of composition provided by version 2 in the future.

wadm

The wasmCloud Application Deployment Manager (wadm) manages Applications. It can be seen as the deployment controller from Kubernetes for wasmCloud Applications. It essentially orchestrates the deployment of components, capabilities, their links, etc. on the platform. This construct will also be dropped with wasmCloud version 2.

Verdict

With a decent understanding of the architecture we can now get an idea of the uses of wasmCloud in the real world. While I have not yet run anything productive on wasmCloud, I have played with the platform a lot over the past few months, and have come to really appreciate some of its innovative ideas.

Thus, to summarise my experience: wasmCloud is a relatively new platform and provides interesting new approaches to how inter-component communication can be modeled. On top of that, it does it while building on open standards such as WebAssembly and the component model, such that the business logic of your application remains portable. While these new concepts are very promising, wasmCloud still suffers from a couple drawbacks:

For people unfamiliar with WebAssembly, it has a quite steep learning curve. This is highly accentuated for people unfamiliar with existing platforms such as Kubernetes.
The set of supported providers and capabilities is extremely small to date. This will of course grow as adoption increases, but currently early adopters will have to write their own providers most of the time and will not be able to rely on third-party components.
As wasmCloud shifts more responsibility to the platform level, it will require a strong platform team to operate this with low developer friction. This can be an issue as finding highly skilled platform engineers is quite difficult at the moment. However, the team behind wasmCloud is focused on making application delivery as frictionless as possible.
Finally, I am not sure I currently understand the security model wasmCloud uses to authenticate and authorize calls between components. While I am not sure this is a drawback, it does not yet feel as intuitive as Kubernetes simple yet relatively powerful RBAC. I will have to dive deeper into this to have a final opinion on it though (another blog post might follow on this).

That Time We Found a Service Account Token in my Log Files

Vincent von Büren — Thu, 04 Sep 2025 07:59:04 +0000

⚠️ Disclaimer
This article assumes you're already somewhat familiar with Kubernetes concepts (Pods, ServiceAccounts) and the basics of JSON Web Tokens (JWTs).

It was a Tuesday.

Nothing special - just your average day as a platform engineer. My team's notifications were mercifully quiet, and I thought, "Perfect, I can finally clean up that old Helm chart that's been bothering me."

I opened the repo of the underlying image written in Go to double-check the config before merging.

Before I even got far, a colleague — Martin Odermatt — pinged me:

“You might want to take a look at this…”

He had spotted something concerning in the code:

log.Println("SA Token:", token)

Wait. What?

A debug statement. Still in production code. Logging an actual Kubernetes ServiceAccount token. Not cool...

I paused. My heart rate didn’t. Curious but mostly horrified, I took the token Martin had found and decoded the payload in my shell:

{
"iss": "https://kubernetes.default.svc.cluster.local",
"kubernetes.io/serviceaccount/namespace": "payments",
"kubernetes.io/serviceaccount/secret.name": "payments-token-6gh49",
"kubernetes.io/serviceaccount/service-account.name": "payments-sa",
"kubernetes.io/serviceaccount/service-account.uid": "f9a2c144-11b3-4eb0-9f30-3c2a5063e2e7",
"aud": "https://kubernetes.default.svc.cluster.local",
"sub": "system:serviceaccount:payments:payments-sa",
"exp": 1788201600,   // Sat, 01 Aug 2026 00:00:00 GMT
"iat": 1756665600    // Fri, 01 Aug 2025 00:00:00 GMT
}

Default audience claim. A 1-year expiry.

As we dug deeper, Martin Odermatt pointed out the underlying issue: this was a legacy ServiceAccount token, and we should be using projected tokens instead.

This "bad boy" wasn't just a dev leftover - it was a high-privilege token with zero constraints floating around in plaintext logs!

What This Article Covers

In this post, I'll guide you through:

The inner workings of Vault authentication with JWT and Kubernetes methods
What Kubernetes ServiceAccounts and their tokens are, and how they’re (mis)used
How projected ServiceAccount tokens fix many of the hidden dangers of older token behavior
Why you should start adopting token projection and Vault integration today

We'll cover real-world use cases, implementation tips, and common pitfalls - so you don't end up like I did, staring at a:

log.Println("SA token:", token)

...and wondering how close you just came to a security incident.

Why This Matters

To really understand why that log statement gave me chills, we need to unpack a few core concepts:

What is a JWT?
How do Kubernetes ServiceAccounts and their tokens work?
And what role do these tokens play in authenticating to systems like Vault?

Let's start with the fundamentals.

What Is a JWT?

If you've been around authentication systems long enough, you've probably seen one of these beasts:

eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...

This is a JSON Web Token (short: JWT).

It's a compact, URL-safe format for representing claims between two parties. They're used everywhere: web apps, APIs, and yes — inside your Kubernetes cluster.

A JWT consists of three parts:

Header – declares the algorithm used to sign the token (e.g. RS256)
Payload – contains the claims (who you are, what you're allowed to do, etc.)
Signature – a cryptographic seal that verifies the payload hasn't been tampered with

Claims are the heart of a JWT — key-value pairs that describe who the token refers to and what it can be used for.

They can be:

Standard claims defined by the spec (e.g., iss, sub, exp, aud)
Custom claims added by the issuer for domain-specific needs

Closer Look at `aud`

The audience (aud) claim tells who the token is meant for. Think of it as the intended recipient.

Example: Imagine a Coldplay concert ticket. It says valid for Stadium X on 01-09-2025. You can't take the same ticket and use it at Stadium Y — they'll reject it (...trust me, I tried).

A JWT works the same way:

If the token has "aud": "https://kubernetes.default.svc", then only the Kubernetes API server should accept it.
If some other service receives that token, the aud won't match and the token must be rejected.

Without this check, a token could be misused anywhere that trusts the signing key. With aud, it's scoped to the right system.

Kubernetes and ServiceAccounts

Kubernetes is an open-source platform that orchestrates containers at scale. At its heart is the Pod — the smallest deployable unit.

But every pod needs an identity. That's where ServiceAccounts come in.

ServiceAccounts 101

Every Pod references a ServiceAccount (default if none is set), but a token is only mounted if enabled
Kubernetes mounts the identity at:

/var/run/secrets/kubernetes.io/serviceaccount/token

That token is a JWT, signed by the Kubernetes control plane
It lets the pod authenticate with the API server — and sometimes even external systems like Vault

The Catch

Until recently, these tokens came with dangerous defaults:

Long-lived (often valid for a year)
Previous to Kubernetes v1.24, there was no default audience set (https://kubernetes.default.svc)
Automatically mounted into every pod, even if unused

Enter Vault: The Gatekeeper of Secrets

HashiCorp Vault is your cluster’s paranoid librarian:
it stores API keys, certs, passwords — and only hands them out when it's sure you should have them.

How? Authentication methods.

Vault Authentication Methods

Username & password
AppRole
LDAP
Kubernetes
JWT

Let's zoom into the last two.

Kubernetes Auth Method

Pod sends its mounted ServiceAccount token to Vault
Vault validates it against the Kubernetes API
If valid, Vault maps it to a policy

This is simple and works well when Vault runs inside the cluster.

JWT Auth Method

Vault verifies the JWT itself (signature, claims, expiration)
No need for Kubernetes API access
More portable

Rule of thumb:

Use Kubernetes if Vault runs inside your cluster and simplicity matters
Use JWT if you want portability, stronger boundaries, and flexibility

Projected Tokens: Because It's 2025

Old tokens were static and long-lived — exactly what we were looking at here. As Martin pointed out during the investigation, projected tokens are designed to fix this entire class of problems.

Instead of mounting a one-year token into every pod, Kubernetes can now generate short-lived, audience-bound tokens on demand.

What You Get

Short TTL (e.g. 10 minutes)
Audience restrictions (aud: vault)
Automatic rotation by kubelet
No automatic mounting into pods

Example Pod with Projected Token

apiVersion: v1
kind: Pod
metadata:
  name: projected-token-test-pod
  namespace: demo
spec:
  serviceAccountName: projected-auth-sa
  containers:
    - name: projected-auth-test
      image: demo/vault-curl:latest
      command: ["sleep", "3600"]
      volumeMounts:
        - name: token
          mountPath: /var/run/secrets/projected
          readOnly: true
  volumes:
    - name: token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 600
              audience: vault

Why Vault Loves This

Vault's JWT auth method is tailor-made for projected tokens:

It parses and verifies the JWT signature (via a configured PEM key or JWKS endpoint)
Validates all claims (aud, sub, exp, iss) locally
Issues secrets only if every check passes

Minimal dependencies. Strong claim validation. Secure, verifiable checks.

Back to the Log

Imagine you stumble upon this in a Go app:

log.Println("Auth Token:", token)

Old world: a one-year, cluster-wide token with no audience. A time bomb.
New world: a 10-minute token, scoped to Vault, rotating automatically.

It's still bad to log tokens — but at least it's not catastrophic.

Try It Yourself: Vault + K8s AuthN Lab

I've built a hands-on demo repo where you can test this locally with KIND (Kubernetes in Docker) and Vault Helm charts:

👉 GitHub: VincentvonBueren/erfa-projected-sa-token

What's Inside

KIND cluster with Vault
Both Kubernetes and JWT auth methods enabled
Vault policies and roles
Four demo pods:
- Kubernetes auth method
- JWT with static token
- JWT with projected token
- JWT with wrong audience (failure demo)

Final Drop 🎤

If your pods still run with default, long-lived tokens:
you’re one debug log away from giving away the keys to your cluster.

Projected tokens aren't optional. They're essential.
Adopt them today — and stop shipping security disasters.

Acknowledgment

The discovery of the exposed ServiceAccount token — and the push towards using projected tokens — came from my dear fellow engineer Martin Odermatt, whose input significantly shaped this investigation and motivated me to tell this story.

Dissecting Kubewarden: Internals, How It's Built, and Its Place Among Policy Engines

Jakob Beckmann — Mon, 07 Jul 2025 05:31:30 +0000

Kubernetes offers amazing capabilities to improve compute density compared to older runtimes such as virtual machines. However, in oder to leverage the capabilities of the platform, these tend to host applications from various tenants. This introduces a strong need for properly crafted controls and well-defined compliance to ensure the tenants use the platform correctly and do not affect one another. The RBAC capabilities provided out of the box by Kubernetes are quickly insufficient to address this need. This is where policy engines such as Kubewarden come into play. In this post we will look at how Kubewarden can be leveraged to ensure correct usage of a platform, how it compares to other policy engines, and how to best adopt it.

Policy Engines

Kubernetes provides role-based access control (RBAC) out of the box to control what actions can be performed against the Kubernetes API. Generally, RBAC works by assigning sets of roles to users or groups of users. Capabilities are attached to these roles, and users having a role obtain these capabilities. This simple mechanism is very powerful, mostly because it is quite flexible while allowing a simple overview of a user's capabilities. However, in the case of Kubernetes, the definition of capabilities is very restricted. Roles only allow or deny access to Kuberenetes API endpoints, but to not allow control based on payload content. This means that these capabilities are mostly restricted to CRUD operations on Kubernetes primitives (e.g. Deployments, Ingresses, or custom resources). Unfortunately, this is often not enough.

For instance, it is quite common to allow users to perform actions on some primitives under specific conditions. An example would be that creating Deploymentss is only allowed as long as its name follows some convention and the pods its creates are not privileged and set proper resource requests/limits. The naming convention cannot be enforced by standard RBAC controls as these have no possibility to represent more complex logic. Controlling the configuration of the pods created by a Deployment is a validation of the payload pushed to the API, and is thus not supported either.

Note: Security contexts and resources on pods can be controlled via methods such as Security Context Constraints or Pod Security Policies and ResourceQuotas. However, these do not reject the creation of the deployment, but will only block the creation of the pods themselves. It is therefore possible to apply a Deployment that is known to not allow the creation of pods. In my personal opinion this is not ideal, as it does not fail early.

These scenarios is where policy engines come into play. They utilise Kubernetes' Dynamic Access Control mechanisms to enable cluster administrators to manage permissions using more complex logic. The exact capabilities of policy engines can vary greatly as these are essentially arbitrary software that validates or mutates Kubernetes requests. However, the majority of major policy engines work similarly. They tend to implement the operator pattern, enabling the configuration of policies using Kubernetes custom resources. In this blog post we will have a look at Kubewarden in more detail, and how it compares to other engines.

Kubewarden Architecture

Kubewarden leverages WebAssembly (WASM) to enable extremely flexible policy evaluation. Essentially, Kubewarden can be seen as a WASM module orchestrator where policies are deployed as serverless functions that get called when necessary. The result of these WASM functions then determines whether an API request against Kubernetes is allowed, denied, or altered (mutated).

This similee can also help explain Kubewarden's architecture. Essentially, the Kubewarden controller (operator) manages policy servers and admission policies. Policy servers can be seen as hosts for the serverless execution of functions, whereas admissions policies are the functions themselves. Therefore, in order to perform policy validation, one needs at least one policy server running to host the policies one wants to enforce. The controller then takes care of configuring the runtime (policy server) to properly run the adequate policy executable with the appropriate inputs when a policy needs to be evaluated. The diagram below illustrates this:

As policies are WASM modules, they can themselves support configuration. This makes policy reuse a major feature of Kubewarden. Complex logic can be contained in the WASM module while exposing some tuning as configuration, allowing a policy to perform a relatively generic task. To understand this better, let us have a look at such a policy:

apiVersion: policies.kubewarden.io/v1
kind: ClusterAdmissionPolicy
metadata:
  annotations:
  name: "cel-policy-replica-example"
spec:
  module: registry://ghcr.io/kubewarden/policies/cel-policy:v1.0.0
  backgroundAudit: true
  mode: protect
  mutating: false
  policyServer: default
  rules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments"]
  settings:
    variables:
      - name: "replicas"
        expression: "object.spec.replicas"
      - name: maxreplicas
        expression: int(5)
    validations:
      - expression: "variables.replicas <= variables.maxreplicas"
        messageExpression: "'the number of replicas must be less than or equal to ' + string(variables.maxreplicas)"
  namespaceSelector:
    matchLabels:
      environment: test

In this example, we are using a WASM module which evaluates a Common Expression Language (CEL) expression to define our policy. Evaluating a CEL expression is not something we want to implement every time ourselves. Thankfully, Kubewarden provides this as a WASM module on their ArtefactHub. Thus we do not need to implement anything and can reuse that module. It is referenced on the module line above. Of course we also need to actually define the CEL expression that should be the heart of the policy rule. This is done within the settings block. Note how we can use object internals (such as replicas defined in a Deployment) in the validation expression. Finally, we need to define on what objects this policy should be evaluated. In order to do this, we provide rules that tell Kubewarden on what Kubernetes API endpoints to trigger the policy, and additionally provide information about which namespaces should be affected by the policy with a namespaceSelector. The remaining options configure the following:

backgroundAudit: informs Kubewarden to report on this policy for objects that are already deployed. In this case, we validate the replicas on created or updated Deployment objects. However, there might already be Deployments on the cluster that violate the policy before we start enforcing it. This option will tell Kubewarden to provide reports on such violations.
mode: Kubewarden supports enforcing policies (in protect mode), or monitoring the cluster (in monitor mode). Using the monitor mode can be interesting when investigating how people use the Kubernetes cluster or providing them with warnings before enforcing policies.
mutating: policies can also mutate (change) requests. In this case we are only performing validation to potentially reject requests. Thus we set mutating to false.
policyServer: as explained above, Kubewarden can manage many policy servers. This simply informs the controller on which policy server this specific policy should be deployed.

As one can see based on the sample policy above, while Kubewarden technically uses programs as policies, it is usually not necessary to write any code to use Kubewarden. This is thanks to its strong focus on module configuration and re-usability. The above CEL module alone already enables the configuration of a very wide range of policies. On top of that, other modules shared on ArtefactHub provide more specific validations or mutations that might incorporate more complex logic. If this is not enough, policy groups (a feature we will not cover in this post) can be utilised to combine other policies and express more complex logic as well. Finally, if one has very specific needs that cannot be addressed by any of the publicly shared modules, one can still fall back to writing code and building ones own module with fully arbitrary logic. How such policies can be written, in actual code, might follow in a separate blog post.

The above architecture of Kubewarden is what makes it stand apart from most other policy engines. Generally policy engines contain the logic fully in the controller, only exposing configuration via the custom resource. Since Kubewarden can essentially execute arbitrary WASM bytecode, it is not bound by the expressiveness of the custom resource declaration.

All this considered, is Kubewarden the best choice for a policy engine and should be used in all scenarios?

Comparison

There are many other policy engines out there, such as Kyverno, Gatekeeper, or Polaris. So why would you choose Kubewarden over any other?

As explained above, Kubewarden provides unprecedented flexibility, thanks to the way it evaluates its policies. This has the massive advantage that you will never reach a point that you have a policy that you would like to enforce but are restricted by the policy engine itself. However, it also has some drawbacks. The primary one being complexity. Writing WASM modules is not for the fainthearted, as WebAssembly is not yet incredibly mature, and most developers will not be familiar with it. The complexity issue can however be sidestepped as the vast majority of policies can be expressed using off-the-shelf WASM modules provided by Kubewarden.

Another aspect that often needs to be considered in enterprise contexts, is support. Kubewarden is an open source project that is loosely backed by SUSE (as it was originally developer for its Rancher offering). Thus enterprise support is only available via a SUSE Rancher Prime. Other tools such as Kyverno are not only more mature, but offer more flexible enterprise support (via Isovalent).

Finally, another aspect to consider is the featureset of a policy engines. Not all policy engines support mutating requests, and are thus much more restricted in their use. However, in this category Kubewarden offers all the features typically desired from policy engines. Some engines such as Kyverno support more features such as synchronizing Secret objects. While this can be useful, it is, in my humble opinion, not a feature for a policy engine.

Of course, there are also personal preference aspects to consider. As an example, Kubewarden and Kyverno handle policy exceptions very differently. Kubewarden has matchers that can be defined as part of the policy itself, which allow to exclude some resources from being validated. Kyverno on the other hand uses a separate CRD called PolicyException. Both have advantages and disadvantages.

Verdict

Kubewarden is a very interesting piece of software. Its internal architecture enables it to be incredibly flexible, at the cost of complexity. However, due to a smart concept of WebAssembly module re-use, that complexity is mostly under the hood, unless one wants or needs to dive deep. In my opinion, Kubewarden can be an absolutely great consideration when ones operates very large Kubernetes clusters what might have quite exceptional requirements. However, even in these cases, I would recommend starting very slow, and slowly building up to the complexity Kubewarden can hold in store.

If you do not operate a large Kubernetes fleet, or expect to have rather standard requirements in terms of how you want to restrict access to you cluster(s), you might be better off with more mature and simpler tools like Kyverno. Getting support for these tools is likely to also be much simpler.

A large part of the complexity of Kubewarden also comes from all that is required to even run this in an enterprise context. Unless you allow pulling WASM modules directly from the internet, you will also need a registry to host OCI packaged modules. On top of that, should you decide to write your own modules, you will need a process to do this, and build knowhow in that area. These are some of the aspects I hope to cover in a follow up post.

Your Cluster Deserves Better Traffic Management. Enter: Gateway API.

Ignacio de los Rios — Thu, 05 Jun 2025 09:07:17 +0000

A Kubernetes cluster is like a beautifully designed city—with excellent plumbing, stable buildings, and all essential services—but no roads leading in or out. No visitors, no deliveries, no exits. If you want your application or services to interact with the outside world (say, users), you’ll need to pave some highways. Kubernetes has traditionally offered a few ways to do this—some better than others. Now, there’s a new infrastructure project in town: the Gateway API—and it's here to stay.
In this post, we’ll explore Kubernetes’ latest external access mechanism and show how it integrates with Cert-Manager to handle certificates. But before diving into this new world, let’s briefly review the most common ways to expose services in Kubernetes.

Current Alternatives

NodePort: Exposing a Service on a Node’s Port

A NodePort service is the most basic way to expose a service externally. Kubernetes opens a specific port on each node, and any traffic to a node’s IP on that port is forwarded to the service inside the cluster. Simple, but with notable limitations.

LoadBalancer: External Load Balancers for Services

A LoadBalancer service goes a step further by integrating with external load balancing infrastructure. When you create a Service of type LoadBalancer, Kubernetes asks the cloud provider to provision an external load balancer (e.g., AWS ELB, GCP Load Balancer, Azure Load Balancer). The service receives an external IP or hostname that forwards traffic to the backing Pods.
This works well in managed cloud environments but scales poorly and doesn’t work out of the box for bare-metal or on-prem clusters—unless you add a software load balancer like MetalLB.

Ingress/Routes: Layer-7 Routing for HTTP/S

Ingress (or Route in OpenShift) is a separate API object used for external access at Layer 7 (HTTP/S). With Ingress, you define routing rules—for example, "send requests for api.example.com to Service A, and www.example.com to Service B."
An Ingress Controller implements these rules, typically using a proxy or cloud load balancer. However, Ingress only supports HTTP(S) and lacks native support for TCP or other protocols.

From Ingress to Gateway API: Why a New API?

The Gateway API is an evolving standard—now an official Kubernetes SIG project—designed to address the limitations of Ingress and support more advanced traffic management. If you're unfamiliar with the term “SIG” it stands for Special Interest Group: volunteer teams that manage and maintain specific areas of the Kubernetes ecosystem (see the project home here). Think of the Gateway API as a modern urban plan for our Kubernetes city — with zoning laws, dedicated roads, and clearly defined roles.

Unlike Ingress, Gateway API is more than a single traffic cop signaling HTTP cars through an intersection. It is an expressway system that speaks multiple protocols—HTTP, HTTPS, TCP, TLS, even UDP—while offering built-in traffic tricks such as path and header rewrites, query-parameter matching, and weighted routing for canary releases - all without using vendor-specific annotations. Crucially, it separates concerns: platform teams lay down GatewayClasses and Gateways (the asphalt), while application teams own their Routes (the traffic signs). Because the spec is vendor-neutral, the very same YAML can drive Istio, NGINX or Cilium, meaning your road network is portable as your underlying architecture evolves.

Three of the most important Resources introduced by GatewayAPI are:

GatewayClass – urban planners
Gateways – the roads and entry points
Routes – detailed traffic control rules

These are implemented as Kubernetes Custom Resource Definitions (CRDs), providing flexibility, scalability, and separation of concerns. Let's dive into each of them.

GatewayClass

A GatewayClass defines a category of Gateways and is managed cluster-wide. It encapsulates the controller responsible for implementing the associated gateways.
This is similar in concept to StorageClass in Kubernetes: administrators define them, and users reference them.
Example:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: istio
spec:
  controllerName: istio.io/gateway-controller

In this case, any Gateway with gatewayClassName: istio will be handled by the Istio controller. It applies similar for other controllers.

Gateway

A Gateway is an instance of ingress infrastructure—an actual network entry point. It references a GatewayClass and defines one or more listeners, each specifying a protocol (HTTP, HTTPS, TCP), port, and optional hostname.
Example:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: example-gateway
  namespace: istio-ingress  # Gateways are namespaced
spec:
  gatewayClassName: istio   # must match a GatewayClass
  listeners:
    - name: http            # an arbitrary name for the listener
      protocol: HTTP
      port: 80
      hostname: "*.example.com"
      allowedRoutes:
        namespaces:
          from: All         # allow Routes from any namespace to bind (could be restricted in production)

Routes: The Core of Gateway API’s Flexibility

Here’s where the Gateway API truly shines—dedicated, protocol-specific route types like HTTPRoute, TCPRoute, and TLSRoute. These give you granular control over traffic, such as:

Matching by host, path, headers, or query parameters
Routing to multiple services with weighted distribution (for canary or A/B testing)

Think of Routes as traffic signs guiding vehicles through the city. They keep everything flowing smoothly.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myapp-route
  namespace: default
spec:
  hostnames:
    - "myapp.example.com"
  parentRefs:
    - name: example-gateway
      namespace: istio-ingress
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: "/"
      backendRefs:
        - name: myapp-service
          port: 80

Example highlights from an HTTPRoute:

parentRefs: Attach the route to a specific Gateway and listener
hostnames: Match specific domains (e.g., myapp.example.com)
rules: Define matching logic and forwarding behavior
matches: Conditions like path prefix or headers
backendRefs: Destination services, possibly weighted

Earlier, we talked about one of the key advantages of the Gateway API over Ingress: native support for advanced traffic management—or what we called “traffic tricks.” A concrete example of this comes in the form of on-object filters. For instance, if you need to strip /v1 from all incoming URLs, here’s how you can do it using Gateway’s built-in URL rewrite filter:

spec:
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: /
    backendRefs:
    - name: myapp-service
      port: 80

No annotations, no sidecars—just one CRD doing the job.

Gateway Controllers and Implementations

Gateway API resources (like Gateway, HTTPRoute, and GatewayClass) are declarative. To actually route traffic, you need a Gateway Controller—a component that reads these resources and configures the data plane (Envoy, NGINX, etc.).
Supported controllers include:

Service meshes like Istio and Linkerd (Istio is moving to Gateway API by default)
Ingress controllers such as Contour, NGINX, Traefik, Kong (adding Gateway support)
Cloud providers (GKE, AWS, Azure) that map Gateway resources to native load balancers
Other projects like Cilium and Gloo offering enhanced networking

Managing TLS Certificates with Gateway API

Let’s be honest: rotating and renewing TLS certificates remains one of the most thankless tasks in day-to-day ops. Exposing HTTP apps to the internet requires HTTPS, and thus, TLS certificates. In the Ingress world, cert-manager automates this via annotations and certificate blocks. Fortunately, Gateway API is supported too.
To automate TLS certificate issuance with cert-manager and Gateway API:

Ensure cert-manager and Gateway API CRDs are installed
Configure a ClusterIssuer or Issuer (e.g., Let’s Encrypt)
Annotate your Gateway to trigger cert-manager Example:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: example-gateway
  namespace: istio-ingress
  annotations:
    cert-manager.io/issuer: "letsencrypt-prod"  # Reference to a cert-manager Issuer or ClusterIssuer
spec:
  gatewayClassName: istio
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "myapp.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: myapp-example-com-tls  # Secret that will hold the TLS cert and key
      allowedRoutes:
        namespaces:
          from: All

Cert-manager will:

Detect the need for a certificate
Create a Certificate resource behind the scenes
Complete the ACME challenge (if using Let’s Encrypt)
Store the key and cert in the referenced Secret
Keep the certificate renewed automatically

This works much like how cert-manager integrates with Ingress—only now, it's applied to Gateway. This seamless integration with cert-manager is key to automating secrets management for external access and eliminates the need for hacky workarounds.

Conclusion

Kubernetes offers several ways to expose services—from the basic NodePort to cloud-managed LoadBalancer and flexible Ingress. The Gateway API builds on those lessons and provides a modern, scalable way to manage external access with better separation of infrastructure and application concerns.
We explored how GatewayClass, Gateway, and Routes work together, and how Gateway controllers like Istio implement them. On the secrets side, tools like cert-manager integrate seamlessly to automate HTTPS. For broader secrets management, tools like HashiCorp Vault are an excellent addition — perhaps a topic for another blog. 😊
Kubernetes has evolved from a small village into a thriving metropolis. And just like any modern city, it needs robust, scalable infrastructure to manage how traffic flows and how identities are verified at its gates. The Gateway API is that next-generation road system—built with urban planning in mind. The pavement is poured; now it’s time to open the on-ramps and let your applications cruise.

The Tortoise and the Hare: do AI Agents Really Help for Software Development?

Jakob Beckmann — Wed, 23 Apr 2025 05:37:56 +0000

Making my development workflow as fast as possible is a big passion of mine. From customizing my development setup to get the last inkling of efficiency out of it, to thinking how to manage notes and knowledge resources to access them as quickly as possible. With the sudden ubiquity of AI in development tools, I came to wonder how AI could help me write code faster. Being quite the skeptic when it comes to AI actually generating code for me (using tools such as Cursor or GitHub Copilot), I came to investigate AI agents which specialise in code reviews. In this blog post I will share my experience using such an agent on a real world case. I will explore where such agents shine and where they are severely lacking.

I am an AI Skeptic

Generally I am not fond of using AI to develop software. My background is mostly in systems software, where correctness of the software can be critical. This means that using tooling that is non-deterministic and might not produce adequate results makes me uneasy. Furthermore, even if AI were to produce amazing results, a developer relying on it could quickly lose understanding of the code. This results in skill atrophy and large risks if the AI reaches the limits of its capabilities. In other words, I am not keen on having any AI generating code for me on a large scale for anything more than a proof of concept or low risk project.

Nonetheless, one would be foolish to ignore AI's capabilities when it comes to developer tooling.

AI Support Agents

Thus starts my journey investigating AI agents that can support me in the software development lifecycle, but whose main use is not to generate code. Many such agents exist, mostly focusing on reviewing code. I am quite the fan of such a use case, as the AI essentially plays the role of another developer I might work with. It reviews my code, provides feedback, suggestions, and potentially even improvements. It however does this immediately after I have opened a pull request, rather than having to wait for days or weeks on a human review.

How is this different from using an AI that generates code you might ask? The main difference lies in the fact that I still have to think on how to solve the problem I am working on, and provide a base solution. This forces me to understand the issue at hand. Thus, I am much better prepared to accept or reject any suggestions from an AI than if the AI just generated a first solution for me. Moreover, people (myself included) tend to be slightly defensive about the code they write. Thus I will, in all likelihood, only accept AI generated code improvements if it offers a real improvement, rather than blindly incorporating them into the codebase.

All in all, it is extremely unlikely that I will lose understanding of the codebase or have my problem solving skills atrophy, but I can iterate on reviews much faster.

CodeRabbitAI

In order to gain first experiences with such an AI agent, I chose to try out CodeRabbitAI. This was not a thoroughly researched decision. The main reason I chose CodeRabbitAI is that I could try it out for free during 14 days and that it integrates well with GitHub. I am aware that performance between AI models varies greatly. However, CodeRabbitAI uses Claude under the hood, a model typically known to perform surprising well on programming tasks. I thus expect it to not perform significantly worse than any other state of the art model out there.

Starting Small

In my opinion, such agents need to be tested on real world examples. One can see demos using AI to generate a dummy web app all over the place. However, common software projects are significantly larger, contain more complex logic, and are less standardized than these demos. Unfortunately, most software I work on professionally is not publicly available, so I cannot use CodeRabbitAI on these. I therefore picked two (still very small) personal projects of mine:

A NeoVim plugin providing a colour scheme.
A command execution engine to run templated commands.

Both projects are extremely small, with under two thousand lines of code. Both projects are written in Lua, a quite uncommon language. I wanted to see how the AI fares against something it is unlikely to have seen too much during its training.

With that in mind, I wrote a first pull request implementing a fix in highlight groups for pop-up menus in NeoVim. I enabled CodeRabbitAI to summarize the PR for me.

The summary looks good, even though it somehow marks some fixes as features. This is especially intriguing as I use conventional commits and explicitly marked these changes as fixes. Additionally, CodeRabbitAI offers a "walkthrough" of the changes made in the PR. In the case of such a simple PR, I found the walkthrough to be mostly confusing. In the case of larger PRs I can however see how this may be appealing.

In reality, I initially opened the PR with only the fixes the pop-up menus. I then pushed commits introducing the support for additional plugins later on. I would have expected CodeRabbitAI to complain that the new commits introduce changes unrelated to the PR, which is not seen as best practice. It did nothing of the sort.

While the summary, walkthrough, and disregard for best practices were unsatisfying, one unexpected benefit emerged: the integration of linting feedback directly within the pull request comments. It provided nitpicks from linting tools (in this case markdownlint. On one side, it is very disappointing to see that the AI agent did nothing more than lint the code and generate a nice comment out of the output. On the other hand it is quite nice that it introduces "quality gates" such as linting without me having to write a pipeline for it. Moreover, producing easily digestible output from a linter is nothing to be underestimated. The quality of life of having this directly as a comment rather than having to go through pipeline logs to read the raw linter output is quite nice. Is it worth two dozen USD per month? No, definitely not!

On the upside, it did update the summary of the PR to reflect the other changes:

The first PR was extremely trivial. It did not introduce any code containing logic. Other than not pointing out that it should probably have been two separate PRs, CodeRabbitAI fared as I would have expected another developer to have reviewed the PR. With two small differences:

The CodeRabbitAI review was close to immediate (took around 30-60 seconds to run). This is amazing to iterate quickly.
Where I would have expected a human reviewer to point our the nitpick or simply approve, CodeRabbitAI is extremely verbose with explanations, walkthroughs, and so on. This in turn wastes time for the author, as he/she would need to read through this. The verbosity could be nicer on larger PRs, but for small concise PRs this is massive overkill and borderline annoying.

To further evaluate CodeRabbitAI's capabilities, I decided to test it on a pull request with more substantial changes...

A More Complex PR

Armed with dampened expectations from my first PR, I opened another PR in the command execution repository implementing a feature affecting multiple files. These changes also update existing logic.

In this second PR, CodeRabbitAI went above and beyond, and generated a walkthrough containing two sequence diagrams showcasing the control flow of the code that was modified! I was actually quite impressed by this. While probably not necessary for the author of a PR, this is great even only for documentation purposes. New team members with less experience may benefit from such visual aids to understand complex logic within the code. Unfortunately the diagrams didn't highlight the specific modifications introduced by the pull request.

However, the supporting text suddenly becomes more relevant when considering such PRs.

On top of that, CodeRabbitAI actually posted interesting comments. It found the odd nitpick here and there, but also found more meaningful potential issues. For instance, I modified a test configuration to use a different shell. CodeRabbitAI identified that this shell is not listed as a dependency anywhere in the repository, and that it would thus not work off-the-shelf. In this case this was only a test file used to parse the configuration and the configured shell did not affect anything, but this is a great finding generally.

I also started conversing with CodeRabbitAI about some changes. Requesting it to give me a suggestion on some configurations. It managed just fine, but did not actually provide these as code suggestions that can be applied, but rather as code blocks in comments, which was a bit disappointing.

Additionally, I decided to try to use CodeRabbitAI's commands feature. This enables ChatOps to control actions taken by CodeRabbitAI. I generated the PR title using one such command. The title turned out generic and not very informative. In CodeRabbitAI's defense, I am quite unsure how I would have named that PR.

I then tried to get it to write docstrings for new functions that were introduced in the PR. It massively misunderstood the request, and created a PR adding docstrings to all functions in the affected files, even ones that already had docstrings... This goes to show that in some cases, it cannot even do what the most junior of all engineers would be capable of doing thanks to a even so tiny dose of common sense. Moreover, it started adding commits with emojis in the title. This goes to show that these AIs are probably not trained much on professional projects.

After that first disaster, with significantly less ambition, I requested it creates a PR to change a small typo. CodeRabbitAI informed me that it created a branch with the changes included, but that it was not capable of creating pull requests. This shocked me, considering it had created its first disaster PR no 10 minutes before.

After another nudge, CodeRabbitAI however did create a PR. It targeted main instead of the branch I was initially using. I guess this is my own fault though for not being specific enough.

Finally, I also tried to get it to update the wording on a commit it did to use conventional commits. Unfortunately it seems that it only has access to the GitHub API and cannot execute any local git commands. It is therefore not able to perform some relatively common operations in the SDLC that are not part of the GitHub API. However, I am guessing this is subject to change relatively soon with the emergence of technologies such as the model context protocol, which would enable it to control external tools such as git.

All in all, I would say CodeRabbitAI did as I would have expected after the first PR. It corrected nitpicks and allowed me to perform some simple actions. Did it deliver a review of the same quality like a senior engineer familiar with the project would have? No. In fact, in order to test this I intentionally implemented a feature that was already present in the repository, while making a couple design decisions that go against most of what the rest of the repository does. CodeRabbitAI neither detected that the logic I was introducing was already present in the codebase, nor did it complain about the sub-optimal design decisions. This goes to show that such agents are still not capable replacing humans with nuanced understanding of the project's history and architectural principles, potentially leading to the introduction of redundant or suboptimal solutions.

Dashboards!

Another feature of AI agents next to the reviews is the analytics capabilities that come with them. In my personal opinion, analytics are important to measure the impact the introduction of such tooling has on the software delivery. CodeRabbitAI provides a couple nice dashboards on how much it is being used, and what kind of errors it helped uncover.

I did not try out CodeRabbitAI for long enough to have any meaningful metrics, but I am confident that the capabilities provided are enough to get a decent understanding of the quality of adoption.

Moreover, CodeRabbitAI supports reporting. This allows to generate reports based on natural language prompts that could be useful for product owners to get insights of changes made to the software over the course of a sprint.

Verdict

While this whole article might seem like a slight rant against such tools, I would in fact wish I could use such tools at work. Not as a replacement for human reviewers, but as an addition to them. For instance, the quite verbose walkthroughs CodeRabbitAI provides can be a very helpful entrypoint to a human reviewer on larger PRs. Moreover, while the quality of the review is insufficient for projects where quality matters, having near instant feedback is amazing.

Finally, as mentioned above, I believe one major selling point of such agents is in the way we humans interact with them. Even if the agent might do little more than execute linters or similar in the background, having the output of these tools in natural language directly as comments in the PRs is not to be underestimated. This is especially true in the age where more and more responsibility is being shifted to developers. With DevSecOps, developers have to understand and act upon the output of all kinds of tools. Presenting this output in a more understandable format, potentially enriched with explanations, can have a significant impact.

Therefore, as a final word, I would actually encourage people to explore such agents to augment their workflow safely, albeit with caution and a clear understanding of their limitations.

f4z3r / gruvbox-material.nvim

Material Gruvbox colorscheme for Neovim written in Lua

Gruvbox Material

A NeoVim colour scheme in pure Lua allowing for highly flexible configuration and customization.

Features | Installation | Usage and Configuration | API Reference

Note

This is a continuation of the original work from WittyJudge https://github.com/WIttyJudge/gruvbox-material.nvim

A port of gruvbox-material colorscheme for Neovim written in Lua. It does not aim to be 100% compatible with the mentioned repository, but rather focuses on keeping the existing scheme stable and to support popular plugins. This colorscheme supports both dark and light themes, based on configured background, and harder or softer contrasts.

Dark theme:

Light theme:

Different contrasts

Contrast	Dark	Light
Hard
Medium
Soft

Features

Supported Plugins:

Please feel free to open an issue if you want some features or other plugins to be included.

…

View on GitHub

f4z3r / sofa

A command execution engine powered by rofi.

Sofa

A command execution engine powered by `rofi` and `fzf`.

About

sofa is a small utility to enable easy execution of templated commands. It can be used to store snippets that you often rely on, or fully template complex commands. It is meant to be used with a shortcut manager to enable launching from anywhere, but can also inject commands into your current shell session for commands that make more sense to run there (see Integration).

Examples

For Snippets Management

You can use sofa for standard snippets management. Use the integration described below, and have configuration such as:

Configuration

namespaces
  lua:
    commands:
      install-local:
        command: luarocks --local make --deps-mode {{ deps_mode }} {{ rockspec }}
        description: Install rock locally
        tags:
        - local
        - luarocks
        interactive: true

…

View on GitHub

A Comprehensive Guide to Managing Large Scale Infrastructure with GitOps

Jakob Beckmann — Tue, 08 Apr 2025 04:31:02 +0000

GitOps is getting adopted more and more. However, there still seems to be some confusion as to what GitOps is, how it differs from regular CI/CD pipelines, and how to best adopt it. In this post we
will quickly cover what GitOps is, and the three main lessons learned from using GitOps to manage infrastructure at scale both on premise and in the cloud.

GitOps Overview

GitOps is a set of principles enabling the operation of a system via version controlled, declarative configuration. More specifically, the OpenGitOps project defines four principles which define whether a system or set of systems is managed via GitOps:

Declarative: A system managed by GitOps must have its desired state expressed declaratively.
Versioned and Immutable: Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.
Pulled Automatically: Software agents automatically pull the desired state declarations from the source.
Continuously Reconciled: Software agents continuously observe actual system state and attempt to apply the desired state.

Note that git is not referenced anywhere, as GitOps is not bound to any tooling. However, in layman terms, many consider a system operated via git to be a GitOps system. This is not quite correct.

GitOps is More than CI/CD Pipelines

Taking the "layman's definition" from above, any system that has CI/CD via pipelines triggered on repository changes would be a GitOps system. This is not accurate. Consider an IaC pipeline which applies declaratively defined infrastructure (such as a standard opentofu apply in a pipeline, or a Docker build followed by a kubectl apply). While such a system adheres to the first two principles, it does not adhere to the latter two. This implies that changes made to the target system are not corrected (reconciled) until the pipeline runs the next time. Similarly, if the pipeline fails for whatever reason, the desired state does not change the pipeline: a configuration drift is not detected, even if not reconciled.

This is an important distinction when considering "standard CI/CD" and GitOps. Simply having something declared as code does not make it GitOps.

The Advantages of GitOps

GitOps has many advantages over standard ways of managing systems. The advantages of having a declarative desired state, version controlling it, and interacting with the system only via git (or whatever version control system you use) are tremendous. From improved security and higher efficiency to better change visibility. These are well known to most people and will thus not be covered here.

Drift detection and automatic reconciliation are the two other aspects that make GitOps absolutely amazing. This is especially true in the current day and age, with the proliferation of complex systems being worked on by many people concurrently. Being able to observe that the system is not in the desired state has massive advantages, such as for standard SRE operations. Continuous reconciliation ensures that manual operational tasks are kept to a minimum, and that systems cannot degrade over time as small undesired changes creep in.

Tooling

In this post we will mostly focus on using GitOps to manage resources handled via the Kubernetes API, but it should be noted that GitOps as a concept is in no way restricted to Kubernetes. In the Kubernetes space there are two major players for GitOps: ArgoCD and FluxCD. We will not go into the details as to what the advantages for each tool are, other than saying that according to our own experience, ArgoCD might be more developer focused, while FluxCD might suit platform engineers with more Kubernetes experience that want more flexibility.

The rest of this post is tool agnostic and everything we are talking about can be done with either tool (but some aspects might be easier to do with one or the other).

Infrastructure: Disambiguation

Before we dive into how to structure your GitOps configuration, it might make sense to draw a line as to where infrastructure starts and where it ends. We consider infrastructure everything that is part of the platform provided to an application team. Hence this line might vary depending on the maturity of the platform you provide your teams. If we consider a simple Kubernetes platform with little additional abstraction for its users, the infrastructure would contain the Kubernetes platform itself as well as all system components that are shared between the teams, such as a central monitoring stack, a central credential management solution, centralized policy enforcement of specific Kubernetes resources, and the like.

The lower end of the spectrum will likely not be managed by GitOps. That is simply because the GitOps tooling itself typically needs to run somewhere, and also needs to be bootstrapped somehow. Some tools such as FluxCD allow the GitOps controller to manage itself, but even in these cases the runtime for the controller needs to exist when the controller is initially installed, and is thus typically not part of the GitOps configuration.

Now that this is cleared up, let us consider how the configuration should be managed.

App-of-Apps

A very popular pattern for managing configuration via GitOps is the "app-of-apps" pattern. This was popularized by ArgoCD, but is also applicable to other tooling. We will use ArgoCD in the example below, but the same can be implemented using FluxCD Kustomizations.

Let us consider a component from our infrastructure that we want to manage via GitOps. Typically, we would need to tell the GitOps controller how to manage this component. For instance, let us assume the component is installed via raw Kubernetes manifests. Then we would tell the GitOps controller which repository contains these manifests and in which namespace to install them. Depending on the controller you are using, you might also configure additional parameters such as how often it should be reconciled, whether it depends on other components, and so on. In ArgoCD jargon this would be an "Application" (the root of "app-of-apps" naming), and would look as follows:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
spec:
  project: default
  source:
    chart: sealed-secrets
    repoURL: https://bitnami-labs.github.io/sealed-secrets
    targetRevision: 1.16.1
    helm:
      releaseName: sealed-secrets
  destination:
    server: "https://kubernetes.default.svc"
    namespace: kubeseal

You would then apply this Application resource to Kubernetes. Your component would then be managed by GitOps, as any changes you push to the manifests repository would be reflected on the Kubernetes cluster.

Then a second infrastructure component needs to be installed, and you repeat the process. The result would be a second Application which installs and manages a component. You might also want to version your deployment (such as using version 1.16.1 of the Helm chart). This implies that lifecycles require a change to this Application manifest, and thus a call against the Kubernetes API to edit it.

The end result is a set of Application resources, some of which you periodically modify when lifecycling a component. Now imagine you need to deploy your infrastructure elsewhere (for instance a second Kubernetes cluster in our example), or maybe even a couple dozen times. Then you need to manage this entire set of Application resources on every platform. A better approach is to add an abstraction layer, which itself deploys the Application resources via GitOps. Hence you put all your Application resources into a repository, and define another, "higher level" Application which deploys this repository. This means that when deploying to new platforms, you only need to deploy that one "higher level" Application, and any changes to the component Application resources can be made via Git, conforming to our GitOps approach. This "higher level" Application is only there to deploy the component Applications thus the name "app-of-apps". Visually, you thus have the following structure:

It should be noted that this also massively helps when customizing platforms. Typically, components cannot be deployed truly one-to-one in several places, but require slight configuration differences. Consider for instance hostnames for UIs of your components. Two of these components deployed in different locations cannot share the same hostname and routing. Using an "app-of-apps" approach allows you to define variables on the top level application, and inject these into the downstream applications such that they can slightly adapt the way they are installed. We will not dive deeper into how this is done as it is highly dependent on the tooling you use (ArgoCD uses ApplicationSet, FluxCD uses variable substitution), but know this is enabled by such an approach.

Consolidating your Configuration

In the organisation I first used GitOps at scale, we deployed all our components as Helm charts to a Kubernetes cluster. Each component was essentially contained within two different repositories in our version control system:

the source code repository which typically built a Docker image as an artefact
the Helm chart definition which referenced the Docker image from above

When we then introduced GitOps, we decided to add a third repository containing the exact deployment definition (in our case the Application declarations) for the component. Using the app-of-apps pattern from above, we could then reference each of these "GitOps repositories" and deploy specific overlays (customizations) of the Application to specific platforms. This worked well for quite some time. However, with time the number of components we managed increased, and so did the number of target platforms to which these components needed to be deployed. This lead to quite a few issues.

When a new target platform was introduced, all such "GitOps repositories" needed to be updated to contain a new overlay customizing the Application to the specific platform. This is very tedious when you have several dozen such repositories.

Moreover, components had dependencies to other components. This meant that we were referencing components within a repository that were defined in another repository. While not problematic in itself, this can become very tricky when one component has a dependency on a configuration value of another component. The configuration value is then duplicated in both repositories and becomes difficult to maintain. While this sounds like we did not properly separate the components, it is very common to see such cases in infrastructure configurations. Consider for instance a deployment of an ingress controller which defines a hostname suffix for its routes. All components deployed on the same Kubernetes platform that deploy a route/ingress will need to use exactly that hostname suffix in order to have valid routing.

The above issue also results in tricky situations when configurations need to be changed for components that are dependent on one another. If the deployment configuration is separated into different repositories, PRs to these repositories need to be synchronized to ensure the deployment occurs at the same time.

Finally, distributing the deployment configuration over so many repositories meant that it became increasingly difficult to have an overview of what is deployed on a target platform. One would need to navigate through dozens of repositories to check this is correctly done.

After identifying these issues we decided to move all our configuration into a single repository. This repository would then contain a templated definition of the entire set of components which would need to be deployed. A set of platform definitions within the same repository would then feed values to templates to ensure consistent configuration. This massively helped us with to address the issues mentioned above. On top of that, it allows to version the "template" and thus enables rollouts of a versioned infrastructure layer.

You can find an example repository of such a structure
designed with FluxCD here:

f4z3r / flux-demo

This repository shows an example how one can use a single mono-repository to manage multiple clusters' infrastructure in a controlled fashion.

Flux Demo

An example how one can use a mono-repo to manage large infrastructure in a controlled fashion using FluxCD.

Setup | Structure of the Repo | Application vs Infrastructure | Workflow

Setup

Generate a GitHub PAT

See the documentation: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens

For fined-grained control, grant the token Admin and content read/write permissions on the repository.

Setup a Cluster with Flux

Setup the required tooling with devbox shell, then

# install a cluster
kind create cluster -n demo
# set the token
export GITHUB_TOKEN='<redacted>'
# onboard flux
flux bootstrap github \
  --token-auth \
  --owner=f4z3r \
  --repository=flux-demo \
  --branch=main \
  --path=clusters/demo \
  --personal

Sample output from Flux installation

► connecting to github.com
► cloning branch "main" from Git repository "https://github.com/f4z3r/flux-demo.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed component manifests to "main" ("158753158f3c760f741f22ed7f68bdee1b66e475")
► pushing component manifests to "https://github.com/f4z3r/flux-demo.git"
► installing components in

…

View on GitHub

Gitops Bridge

The last challenge we want to address in this blog post is a concept called a "GitOps bridge". In public cloud environments, there is typically a relatively strong cut between infrastructure deployed via Terraform (or any similar tool), and the infrastructure deployed via GitOps. For instance, one might deploy an Azure Kubernetes Service and some surrounding services (such as the required network, a container registry, etc) via Terraform, and them deploy components and applications within the AKS using GitOps. The issue that we face here is that the GitOps configuration very often depends on the Terraform configuration. Consider for instance the container registry. Its address is set up by Terraform, but is used in every image declaration in the GitOps configuration. One option is to duplicate such values in the respective configurations, while another option is to use a GitOps bridge.

The GitOps bridge is an abstract concept on how to pass configuration values from tooling such as Terraform as inputs to the GitOps configuration. How this is done in practice very much depends on which tools you use. For instance, if looking at Terraform and FluxCD, a common way to achieve this is to have Terraform write a ConfigMap onto the AKS where the FluxCD controller will run containing all variables (and their values) that will be required by the GitOps configuration. The FluxCD controller then supports injecting variables from a ConfigMap via variable substitution.

Using a GitOps bridge has the advantage that changes in the Terraform configurations are much less likely to break the GitOps configuration that builds on top of it. Moreover, it allows Terraform to directly bootstrap the entire GitOps setup when creating new platforms without the need to manually redefine the required variables in the GitOps repository.

Summary

So, to recap, we have looked at what GitOps really is (and isn't). Understanding these basics is critical to correctly implement GitOps in your projects. On top of that, we looked at three best practices:

Use an app-of-apps pattern to improve resiliency for when you need to recreate platforms.
Consider using a mono-repository for all your GitOps configuration as your setup grows.
Have a look at GitOps bridges to improve the automation when setting up platforms and ensuring your Terraform and GitOps configurations are consistent.

I hope this has helped you understand a bit better how to use GitOps at scale. If you have any questions or comments, feel free to let me know below.

How to Join Lists in Kafka Streams Applications

Jan Kleine — Mon, 17 Mar 2025 06:00:00 +0000

I recently joined a project where we do data processing with Kafka Streams applications and came across an interesting problem: "list joins". There was already a working solution to the problem, but I was not quite satisfied, so I dug a little deeper. Since I didn't find much on the topic online, I wanted to share my findings here.

TL;DR The final topology is discussed here and you can find the code here:

iptch / kafka-list-join-demo

A demo showing how to do a list join in a kafka streams application.

Kafka List Join Demo

This project shows two ways to perform a list join in a Kafka streams application. A list join refers to joining a record that contains a list with a KTable, such that each element in the list gets joined with the corresponding element in the KTable.

This image shows the high level idea, joining a persons address list, with the corresponding addresses:

Check out this blog post for a discussion of the approaches.

The tests should cover all relevant cases of message ordering and updates.

Building

To build and test the project, run

./gradlew clean build

View on GitHub

I assume you have some understanding of Kafka Streams, but I'll try to link to relevant documentation and resources where possible. It's worth checking out the Kafka Streams DSL Developer Guide if you are unfamiliar with Kafka Streams.

The Problem

We have a topic with records (of type Outer, e.g., a person) that contain (among other things) a list. The elements in the list reference records on another topic (of type Inner, e.g., addresses). For every element in the outer topic we want to look up the corresponding record in our inner topic, and merge them to enhance our list.

Additional Considerations

There are a few additional constraints we need to keep in mind:

Eventual Consistency: The corresponding records on the inner topic may not be available right away, that is, records may come in after our outer record.
Updates: Our outer record may be updated over time. This may include updates (additions/removals) of inner list elements, but also updates to fields other than the list.
Duplicates: In our specific use case we don't need duplicate list entries. However, all approaches discussed here can be modified to allow duplicates without too much effort.

The Existing Solution

The existing solution to the problem looks roughly like follows:

We set a timestamp on every record on our outer stream (needed later).
We flat map our outer records so we have exactly one inner list element in every flat mapped record, to differentiate the flat records we change the keys to composite keys (of the form <outer-key>$$<inner-key>).
We interpret the resulting stream as a KTable and perform a foreign key left join with the other KTable. The KTable-KTable left join (as opposed to, e.g., a KStream-KTable join) is needed to fulfill our eventual consistency constrain.
We group the the resulting records by the first part of the composite key.
Finally, we reduce the records in each group by appending the lists, but we only consider the newest timestamp we see. Whenever we see a newer timestamp we discard all older records. This ensures that we do not add stale elements to our list that may be deleted in the current list.

The code looks similar to the below, though I left some smaller details out, such as how we forward tombstones. The full code can be found here.

(outerKStream, innerKTable) -> outerKStream
    .mapValues(outerMapper)
    .flatMap(outerFlatMapper)
    .toTable(buildStore(outerSerde, "listJoinFlatStore"))
    .leftJoin(
            innerKTable,
            outer -> outer.getInnerCount() == 0 ? null : outer.getInner(0).getId(),
            outerInnerJoiner,
            buildStore(outerSerde, "listJoinJoinerStore")
    )
    .toStream()
    .groupBy(
            (key, _) -> key.split("\\$\\$")[0],
            Grouped.with(Serdes.String(), outerSerde)
    )
    .reduce(
            outerReducer,
            buildStore(outerSerde, "listJoinReducerStore")
    )
    .toStream()

Issues

While this approach works for our use case, I don't like it for two reasons:

It feels hacky: The timestamps feel a bit hacky and force us to change our data model just for this operation. While we can (and in production do) clear the timestamps after the list join is complete, we are using protobufs for our records. So the timestamp is part of the protobuf definition, even if downstream services don't need it.

It is inefficient: Whenever the original protobuf changes (be it, a change to the list, or any other field), all list elements go through the join again, followed by reducing them all (including stale old records) again. That just feels wasteful.

These issues prompted me to spend some time on the problem and see if I can come up with something better.

Improvements

We want to reduce the number of join and reduce operations and remove the need for a dedicated timestamp field.

For this we need a component that keeps track of the changes in the list of a record so that we only process relevant changes.

The List Join Pre-Processor

The only way I found to accomplish this, is to write a custom Processor using the Processor API provided by Kafka Streams.

The list join pre-processor performs the task of the flat-map operation from before, but maintains internal state¹ to remember what list elements are currently in each record. When a record gets updated, it compares the previous and new list, and (1) only issues flat mapped records for new list elements and (2) issues tombstones for elements removed from the list.

(1) ensures that we only have to join new list elements, reducing duplicate work. (2) allows us to more efficiently remove old list elements, so we don't need to rely on a hacky timestamp.

The pre-processor also always forwards a copy of the current record with an empty list. We need this later to correctly propagate changes to other fields that are not the list.

The logic of the pre-processor, which can also be found here, looks as follows:

public void process(Record<String, TOuter> record) {

    // in case of tombstone or empty list, this map is empty
    Map<String, TOuter> flatValuesMap = flatMapper.apply(record.value()).stream()
            .collect(Collectors.toMap(
                    innerIdStringExtractor,
                    Function.identity(),
                    (_, newValue) -> newValue
            ));

    Set<String> newIds = flatValuesMap.keySet();

    Set<String> oldIds = Optional.ofNullable(listStore.get(record.key()))
            .map(Set::copyOf)
            .orElse(Set.of());

    // if both new and old ids are empty we don't need to do anything and can short circuit
    if (newIds.isEmpty() && oldIds.isEmpty()) {
        return;
    }

    // forward flat mapped records for new list elements
    Sets.SetView<String> addedIds = Sets.difference(newIds, oldIds);
    for (String newId : addedIds) {
        forward(record.key(), newId, flatValuesMap.get(newId));
    }

    // send tombstones for removed list elements
    Sets.SetView<String> removedIds = Sets.difference(oldIds, newIds);
    for (String removedId : removedIds) {
        forward(record.key(), removedId, null);
    }

    // if the current list is empty delete the list from the store and send a tombstone
    // for the empty list record, otherwise save the current list and send an empty list
    // record to propagate changes to other fields
    if (newIds.isEmpty()) {
        forward(record.key(), null, null);
        listStore.put(record.key(), null);
    } else {
        forward(record.key(), null, listCleaner.apply(record.value()));
        listStore.put(record.key(), List.copyOf(newIds));
    }
}

Better Reduce Operation

Now that the flat-map and the subsequent join are improved, we can further improve the reduce operation.

Previously, the reduce operation was responsible for filtering out outdated records (based on the timestamp) and aggregating the list of inner elements. This was done on all flat-mapped values (including outdated values) and needed to happen every time anything in the original outer record changed.

Now, that we can issue tombstones for removed elements, we can transition from a KStream group-by and reduce to a KTable group-by and reduce.

The KTable reduce works a bit differently than the KStream reduce. Where the KStream reduce has a single reducer that receives the current aggregate and any new record (be it a tombstone or a normal record), the KTable reduce has an adder reducer and a remover reducer.

The adder reducer is called when ever an element is added to the KTable and receives the current aggregate and the newly added record.

The remover reducer is called when ever an element is removed from the KTable (via a tombstone) and is provided with the current aggregate and the removed record. This allows us to remove the inner element from the current aggregate, as we know exactly which one got removed.

Our adder looks as follows. Remember that our pre-processor also sends a copy of the original record with an empty list. This way we can update other fields as well.

public Outer apply(Outer currentValue, Outer newValue) {
    if (newValue.getInnerList().isEmpty()) {
        // if list is empty update all fields other than the list
        return newValue.toBuilder()
                .addAllInner(currentValue.getInnerList())
                .build();
    } else {
        // otherwise add inner item to current list
        return currentValue.toBuilder()
                .addAllInner(newValue.getInnerList())
                .build();
    }
}

The full code can also be found here.

Our remover only has to worry about removing the list elements. In our case we do it based on the inner ID.

public Outer apply(Outer currentValue, Outer oldValue) {
    // if oldValue inner list is empty, there is nothing to do
    // the adder handles empty list updates
    if (oldValue.getInnerList().isEmpty()) return currentValue;

    // remove the single inner value from the current list,
    // in this case we do it by id
    Inner innerToRemove = oldValue.getInnerList().getFirst();

    List<Inner> innerList = currentValue.getInnerList().stream()
            .filter(inner -> !inner.getId().equals(innerToRemove.getId()))
            .toList();

    return currentValue.toBuilder()
            .clearInner()
            .addAllInner(innerList)
            .build();
}

The full code can also be found here.

New Topology

Incorporating the two improvements from above, we end up with a topology which looks surprisingly similar to our initial topology.

(outerKStream, innerKtable) -> outerKStream
    .process(preProcessorSupplier)
    .toTable(buildStore(outerSerde, "listJoinFlatStore"))
    .leftJoin(
            innerKTable,
            // if the inner list is empty the foreign key extractor should
            // return null so the outer is joined with null
            outer -> outer.getInnerCount() == 0 ? null : outer.getInner(0).getId(),
            outerInnerJoiner,
            buildStore(outerSerde, "listJoinJoinerStore")
    )
    .groupBy(
            // group by first part of composite key
            (key, value) -> KeyValue.pair(key.split("\\$\\$")[0], value),
            Grouped.with(Serdes.String(), outerSerde)
    )
    .reduce(
            listAdder,
            listRemover,
            buildStore(outerSerde, "listJoinReducerStore")
    )
    .toStream()

Again, I skipped over some smaller details, like how we forward tombstones. The full topology can be found here.

Since copying this code to different topologies is a bit cumbersome, we decided to wrap this entire sub-topology in a ListJoin utility. This allows easier reusing of the code, as the developer only has to provide a couple key components and make the resulting topologies easier to understand.

// setup
ListJoin<...> listJoin = ListJoin.builder()
        // specify joiner, reducers, etc.
        .build()

// later in the topology
KStream<...> joinedKStream = listJoin.apply(myLeftKStream, myRightKTable);

Conclusion

This topology is probably still not optimal. For one, it may make sense to deduplicate the empty list records to further reduce downstream work. However, in our case this is not necessary.

In any case, this new approach is a clear improvement over the previous timestamp based approach.

Have you solved a similar, or even the same, problem before? If so, please consider leaving a comment as I'd very much like to hear what you did. Likewise, if you have found issues with the above code or have ideas for improvements, I'm keen to hear from you!

Acknowledgments

Cover image: "Close Up Photo of Blue Background" by Harrison Candlin

Jan KleineFollow

Maintaining state can be done via a state store, which is automatically backed up to a changelog topic in kafka, ensuring the processor can tolerate application scaling and recover in case of application failures. ↩

Data Governance with dbt, Terraform, and Dataplex: A Practical Guide to BigQuery Policy Tags

Diana Tahchieva — Mon, 24 Feb 2025 07:39:48 +0000

Welcome to a hands-on guide for implementing BigQuery Policy Tags, an important feature for data governance. If you're new to Google Cloud Platform (GCP) and have heard of dbt, Terraform, and Data Catalog but aren't sure how they work together, this tutorial provides a simple, practical example. We'll apply policy tags to a sample clients table in BigQuery to enforce data governance.

What Are Policy Tags?
Policy Tags are classification labels for data in BigQuery, helping manage privacy, compliance, and access control. These tags are particularly important in industries like healthcare and finance, where data sensitivity is a key concern.

Why Use dbt, Terraform, and Dataplex for Policy Tag Management?
Using dbt and Terraform to define Policy Tags as code, and Dataplex for governance, you can keep track of changes, facilitate team collaboration, audit activities, and easily roll back to previous configurations.
Additionally, you can consistently manage Policy Tags across multiple datasets and projects, reducing manual labor. Automation through these tools minimizes human error and ensures policy tags are consistently applied.
While dbt integrates seamlessly into existing data pipelines, applying Policy Tags during data transformation and modeling, dataplex unifies governance across various data stores.

Understanding the Tools

Let us have a closer look at what are dbt, terraform and dataplex.
• dbt (Data Build Tool) is an open-source tool for transforming and modeling data within your data warehouse (like BigQuery). dbt enables you to write a more maintainable SQL code and allows you to attach metadata, such as Policy Tags, to your data transformations. Additionally, as the objects in BigQuery can be referenced, dbt is able to make a Directed Acyclic Graph of the entire data platform, based on which we can observe all dependencies among the data.
• Terraform is an Infrastructure as Code (IaC) tool that lets you define and manage your cloud infrastructure using configuration files. Terraform automates the provisioning and management of resources including enabling APIs, managing permissions, and creating Policy Taxonomies and Tags.
• Dataplex is a Google Cloud service that provides unified data governance and management. It helps discover, organize, and manage data assets, ensuring consistent data handling and enforcement of Policy Tags.

How They Work Together

OK, now that we know what dbt, Terraform, and Dataplex are, let's explore how they work together.
Terraform establishes the required infrastructure and permissions for managing Policy Tags by creating taxonomies and tags within Google Cloud Data Catalog. At the same time, dbt handles data transformation in BigQuery, applying Policy Tags to specific columns within your models. The meta section in the dbt model facilitates metadata association, ensuring proper organization and governance. Meanwhile, Dataplex functions as a centralized governance layer, maintaining consistency in the application and monitoring of Policy Tags across all data assets. Together, these tools create a seamless, scalable, and automated data governance system that enhances visibility, reduces manual effort, and minimizes the risk of human error.

Implementing Policy Tags: Step-by-Step Guide

First, you need to create a GCP project and connect it to a billing account; otherwise, Terraform won’t be able to function. Don’t worry—this project is small and won’t incur any costs (my billing account is still at zero). Just remember to delete it after testing by running terraform destroy in the command line—I’ll remind you at the end of the tutorial.
For convenience, I’ve created a Git project that you can clone. It contains two sub-projects: one for Terraform (gcp-data-catalog-terraform) and one for dbt (data_catalog_dbt_project). In a real-world scenario, these sub-projects would likely be managed as separate projects.

The terraform project has the following structure:
• variables.tf: Defines the GCP project ID and region.
• iam.tf: Creates service accounts for Terraform and dbt, assigning necessary IAM roles.
• datacatalog.tf: Defines the taxonomy for organizing Policy Tags and creates tags for PII and non-PII data.
• bigquery.tf: Creates a BigQuery dataset and a table without predefined policy tags.
• output.tf: Outputs IDs for easy access to created resources.

The dbt project in this example is a significantly simplified version of a standard dbt project and includes only the essential files necessary for our use case. Here’s a brief description of each file and its purpose:
• dbt_project.yml: This is the primary configuration file for your dbt project. It includes metadata about the project, such as the project name and version, paths to model files, and configurations for materializations and other project-wide settings.
• profiles.yml: This configuration file contains the connection details and credentials required for dbt to connect to your data warehouse (BigQuery in this case). It includes information such as the project ID, dataset, and authentication method (service account key file).
• models/customers/: This directory holds the models for the project. In dbt, a model is essentially a SQL file that transforms raw data into more refined tables. This directory contains our specific model for customers.
o customers.sql: This SQL file represents the transformation logic for the customers’ data. It selects and processes the necessary columns from the raw data, applying the transformations required for our data analysis needs.
o customers.yml: This YAML file provides additional metadata about the customers model. It includes descriptions of each column, tests to ensure data quality, and policy tags to enforce data governance rules.

Step 1: Enable APIs with Terraform

First, enable the necessary Google Cloud APIs for your project using Terraform. These are:
• Identity and Access Management (IAM) API
• BigQuery
• Data Catalog API
• Dataplex API

Step 2: Grants Permissions with Terraform

Let us walk through how to create and configure a service account that Terraform and dbt can use to interact with Google Cloud resources.
A service account in Google Cloud is like a robot user—it allows Terraform and dbt to authenticate and interact with GCP without needing a human user to log in.
In our setup, Terraform will provision BigQuery datasets, tables, and policies, while dbt will query and transform the data. To ensure that the permissions for Terraform and dbt are properly separated, we will define two distinct service accounts.

We can create the new service accounts (called terraform-sa and dbt-sa) by running:
gcloud iam service-accounts create terraform-sa --display-name "Terraform Service Account" --project gcp-data-governance
gcloud iam service-accounts create dbt-sa --display-name "dbt Service Account" --project gcp-data-governance

Alternatively, you can add the service account manually in the GCP console (IAM & Admin > Service Accounts). Then, in Terraform, you only manage the IAM roles in iam.tf

The Terraform service account needs permissions to manage BigQuery, Data Catalog, and Dataplex. Add these IAM roles in your Terraform configuration:

# Give the Terraform user permissions to manage IAM, Dataplex, and BigQuery
resource "google_project_iam_member" "terraform_bigquery_admin" {
  project = var.project_id
  role    = "roles/bigquery.admin"
  member  =  "serviceAccount:terraform-sa@gcp-data-governance.iam.gserviceaccount.com" 
}

resource "google_project_iam_member" "terraform_datacatalog_admin" {
  project = var.project_id
  role    = "roles/datacatalog.admin"
  member  = "serviceAccount:terraform-sa@gcp-data-governance.iam.gserviceaccount.com"
}

resource "google_project_iam_member" "terraform_dataplex_admin" {
  project = var.project_id
  role    = "roles/dataplex.admin"
  member  = "serviceAccount:terraform-sa@gcp-data-governance.iam.gserviceaccount.com"
}

resource "google_project_iam_member" "terraform_sa_admin" {
  project = var.project_id
  role    = "roles/iam.serviceAccountAdmin"
  member  = "serviceAccount:terraform-sa@gcp-data-governance.iam.gserviceaccount.com"
}

Note: For more information on the available roles, refer to the documentation. As a best practice, always assign the minimum permissions necessary. For instance, if a user only needs to view the data, provide read-only access.

• To allow dbt to authenticate, we need to generate a JSON key file for our service account, which we can do in two ways:

Option 1: Using the Google Cloud Console

Go to IAM & Admin → Service Accounts in the Google Cloud Console.
Select the service account dbt-sa.
Navigate to the Keys tab and click "Add Key".
Choose "JSON", then click "Create".
The key.json file will be downloaded to your computer.

Option 2: Using the gcloud CLI

If you prefer using the command line, run:

gcloud iam service-accounts keys create dbt-sa-key.json  --iam-account=dbt-sa@your-project-id.iam.gserviceaccount.com

• Now that we have our dbt-sa-key.json, we need to update dbt's configuration to use the service account. Open profiles.yml and update the value of your keyfile:

data_catalog_dbt:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: service-account
      project: "gcp-data-governance"  # project_id
      dataset: multiplexer_dataset
      threads: 4
      keyfile: "/path/to/dbt-sa-key.json"  # Reference your environment variable
      location: "europe-west6"  # Set this to your BigQuery region

Step 3: Create Data Policy Taxonomies and Tags with Terraform

To allow Terraform to interact with Google Data Catalog we need to specify providers. The providers allow us to manage resources on a specific cloud platform. We have created them in datacatalog.tf as they are relevant to Data Catolg API, however you can make a separate file providers.tf and define them there. The google provider is the standard Terraform provider for managing GCP resources, while the google-beta provider gives access to Google Data Catalog, which is needed for policy tags.

Why Not Just Use google-beta?

Some resources (e.g., IAM, BigQuery datasets) don’t need google-beta, so we keep google for those. On the other hand side Data Catalog resources require google-beta, so we configure both providers.

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 4.0"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

provider "google-beta" {
  project = var.project_id
  region  = var.region
}

A taxonomy is a container containing data policy tags ( a grouping mechanism). Fine-Grained Access Control specified in the activated policy tags ensures that only authorized users can see or query certain columns.

# Create a Data Catalog Taxonomy
resource "google_data_catalog_taxonomy" "multiplexer_pii_taxonomy" {
  provider = google-beta
  display_name = "Multiplexer PII Taxonomy"
  description  = "Taxonomy for sensitive data classification"
  project      = var.project_id
  region       = var.region
  activated_policy_types = ["FINE_GRAINED_ACCESS_CONTROL"]
}

In our test-case we define only two tags: sensitive (PII) and non-sensitive data. The PII tags ensure proper access control. These tags will later be attached to BigQuery columns in dbt. More tags can be added as needed for better governance.

# Create Policy Tags for PII and Non-PII
resource "google_data_catalog_policy_tag" "pii_sensitive" {
  provider = google-beta
  taxonomy     = google_data_catalog_taxonomy.multiplexer_pii_taxonomy.id
  display_name = "PII"
  description  = "Policy tag for Personally Identifiable Information"
}

resource "google_data_catalog_policy_tag" "non_pii_sensitive" {
  provider = google-beta
  taxonomy     = google_data_catalog_taxonomy.multiplexer_pii_taxonomy.id
  display_name = "Non-PII"
  description  = "Policy tag for non-sensitive data"
}

Step 4: Apply Policy Tags with dbt

Now that the Policy Tags are defined, we will attach them to relevant columns in the dbt project:

Set up your dbt project in a directory parallel to your Terraform project. Before running dbt, ensure that your service account key is correctly set (see explanation in Step 2). Remember, dbt profile.yml should be configured to use BigQuery and your service account key. Before running dbt, make sure all dependencies are installed:

dbt deps –upgrade

Afterwards check if dbt can connect to BigQuery, by running:

dbt debug

If successful, you’ll see the message:

All checks passed!

To run all dbt models use dbt run and to run only specific models, e.g., customers use dbt run -s customers

Define columns and attach Policy Tags in your dbt models.

Navigate to your dbt project and modify your customers.yml to attach Policy Tags:

version: 2

models:
  - name: customers
    description: "Customers data with PII"
    meta:
    columns:
      - name: customer_id
        description: "Unique customer identifier"
        policy_tags:
          - "{{ var('non_pii_sensitive_policy_tag_id') }}"
      - name: email
        description: "Customer email"
        policy_tags:
          - "{{ var('pii_sensitive_policy_tag_id') }}"
      - name: phone_number
        description: "Customer phone number"
        policy_tags:
          - "{{ var('pii_sensitive_policy_tag_id') }}"

Now run dbt to apply the Policy Tags to BigQuery automatically: dbt run -s customers

Conclusion
Integrating dbt, Terraform, and Dataplex allows you to efficiently manage BigQuery Policy Tags, enforcing data governance policies in a scalable and automated way. This approach enhances security, compliance, and operational efficiency while reducing manual effort.

To avoid unnecessary charges, remember to destroy your project after testing by running terraform destroy in the command line.

Rusty Backends

Jan Kleine — Tue, 21 Jan 2025 06:30:00 +0000

This post was written together @sekael and @zkck

Rust has been a developer favorite for many years now and is consistently highly admired by developers[1]. It made a buzz when it was introduced as a complementary language to C and Assembly in the Linux kernel with version 6.8[2] and its speed and memory safety make it a popular choice for low-level and performance-critical applications. But with the whole world using the internet, we wanted to find out whether it can also do web and challenge the reigning backend champions like Java Spring or Go.

To find answers, we wanted to get our hands dirty with three popular Rust web frameworks including rocket, axum, and actix, and get a feeling for their performance, features, and most importantly the developer experience.

In this post, we will examine each of these frameworks by implementing the same example API endpoints in each one, connect to a MongoDB database, and run the server in a Docker container.

All three, rocket, axum, and actix, cover the full range of functionality you would expect from a web framework, like routes, handlers, request and response parsing, middleware, state management, database interactions, logging, and testing. Furthermore, all of these frameworks perform well and will meet and surpass the needs of most web applications, so the choice really comes down to ergonomics and how it feels to develop within each framework.

So let’s jump right in…

Rocket (rocket.rs)

rwf2 / Rocket

A web framework for Rust.

Rocket

Rocket is an async web framework for Rust with a focus on usability, security extensibility, and speed.

#[macro_use] extern crate rocket;

#[get("/<name>/<age>")]
fn hello(name: &str, age: u8) -> String {
    format!("Hello, {} year old named {}!", age, name)
}

#[launch]
fn rocket() -> _ {
    rocket::build().mount("/hello", routes![hello])
}

Visiting localhost:8000/hello/John/58, for example, will trigger the hello route resulting in the string Hello, 58 year old named John! being sent to the browser. If an <age> string was passed in that can't be parsed as a u8, the route won't get called, resulting in a 404 error.

Documentation

Rocket is extensively documented:

Overview: A brief…

View on GitHub

Rocket is a web framework for Rust that comes with the most batteries included out of the three we looked at. It positions itself in the same league as Rails (Ruby) or Flask (Python) and aims to offer functionality for anything a web backend might need, including route handling, request and response parsing, database integrations, validation, and so on. One of its main goals is to minimize the amount of boiler plate code you will have to write, and it does so through metaprogramming, i.e. heavy use of Rust macros.

This makes it easy for the developer to integrate her actual logic with the rocket framework. Let’s have a quick look on how you might define a route handler including parameter guards and validation, a database connection pool, and a JSON object.

We can define a route handler simply by adding the appropriate macro to a function.

#[get("/texts/<uuid>")]
pub async fn get_text(db: Connection<TextsDatabase>, uuid: Uuid) -> (Status, Value) {
    // ...
    (Status::Ok, json!({"data": data}))
}

If the request parameter cannot be parsed, the appropriate response code is sent instead of calling the function, so we have the usual comfort of type safety. Request bodies are handled very similarly, and of course, parsing integrates seamlessly with serde.

#[derive(Deserialize)]
pub struct Message<'m> {
    pub data: &'m str,
}

#[post("/texts", format = "application/json", data = "<msg>")]
pub async fn post_text(db: Connection<TextsDatabase>, msg: Json<Message<'_>>) -> (Status, Value) {
    // ...
}

Adding a database is as simple as annotating the appropriate struct and initializing it at startup. Of course many popular DBMS's are supported.

#[derive(Database)]
#[database("texts")]
pub struct TextsDatabase(mongodb::Client);

Rocket goes heavy on the nerdy space vocabulary - which adds to its charm. Thus, to get your web server off the ground, you simply launch it. Notice again, how annotations are used to get the rocket going.

#[launch]
fn rocket() -> _ {
    rocket::build()
        .attach(TextsDatabase::init())
        .mount("/", routes![get_text, post_text])
}

We have found rocket very enjoyable to work with, especially if you want to focus on the business logic of your server and feel safe trusting the framework to take handling of the boilerplate code off your hands. If you are rather new to Rust and maybe switching over from other frameworks that make heavy use of annotations, you will feel at home with Rocket.

Axum

tokio-rs / axum

Ergonomic and modular web framework built with Tokio, Tower, and Hyper

axum

axum is a web application framework that focuses on ergonomics and modularity.

More information about this crate can be found in the crate documentation.

High level features

Route requests to handlers with a macro free API.
Declaratively parse requests using extractors.
Simple and predictable error handling model.
Generate responses with minimal boilerplate.
Take full advantage of the tower and tower-http ecosystem of middleware, services, and utilities.

In particular the last point is what sets axum apart from other frameworks axum doesn't have its own middleware system but instead uses tower::Service. This means axum gets timeouts, tracing, compression, authorization, and more, for free. It also enables you to share middleware with applications written using hyper or tonic.

Usage example

use axum::{
    routing::{get, post},
    http::StatusCode,
    Json, Router,
};
use serde::{Deserialize,

…

View on GitHub

If you are looking for a framework that is more bare-metal Rust and comes with less batteries included, you may want to check out the axum framework developed by tokio-rs, the same team that is responsible for the tokio async library. Opting for axum means you will have to take care of some more of the boilerplate code but you are also not fighting a potentially opinionated framework to get your logic just the way you want it. Compared to rocket, axum does not depend heavily on macros or annotations, and it sets itself apart from its peers by not implementing its own middleware but rather relying on tower for this. Through the tower ecosystem axum can offer timeouts, tracing, compression, authorization, and much more, while also enabling you to share your middleware with applications written with other web libraries like hyper.

Looking at axum’s ergonomics, previous Go developers will feel right at home. The syntax looks very similar to a web server written e.g. with the gorilla/mux module in Go. So how would you go about the implementation of your first route handler including connecting to a database and starting the server? Let’s have a look.

The same two functions from above look something like this:

async fn post_text(
    State(state): State<Arc<state::MongoAppState>>,
    Json(text_payload): Json<payloads::TextPayload>,
) -> Result<
    (StatusCode, Json<payloads::InsertedResponse>),
    (StatusCode, Json<payloads::ErrorResponse>),
> {
    // ...
}

async fn get_text(
    State(state): State<Arc<state::MongoAppState>>,
    Path(text_id): Path<String>,
) -> Result<Json<payloads::TextPayload>, (StatusCode, Json<payloads::ErrorResponse>)> {
    // ...
}

No macros, but a lot more verbose. We see the same when we look at the startup code, we have to do a lot more manually, e.g., pass the DB connection along.

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = mongodb::Client::with_uri_str("mongodb://...").await.unwrap();

    let shared_state = std::sync::Arc::new(state::MongoAppState::new(client));

    // build our application with a single route
    let app = Router::new()
        .route("/texts", post(post_text))
        .route("/texts/:text_id", get(get_text).delete(delete_text))
        .with_state(shared_state);

    // run our app with hyper, listening globally on port 3000
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await.unwrap();
    axum::serve(listener, app).await.unwrap();
    Ok(())
}

Axum is a very powerful web framework that benefits from the expertise of the tokio-rs team in creating great Rust tooling. It is being very actively developed and relies on other state-of-the-art crates for middleware. You will most likely enjoy axum the most if you are already familiar with Rust, enjoy having control over a lot of detail in implementation, or maybe if you are switching from another low-level web language like Go.

Actix (actix.rs)

actix / actix-web

Actix Web is a powerful, pragmatic, and extremely fast web framework for Rust.

Actix Web

Actix Web is a powerful, pragmatic, and extremely fast web framework for Rust

Features

Supports HTTP/1.x and HTTP/2
Streaming and pipelining
Powerful request routing with optional macros
Full Tokio compatibility
Keep-alive and slow requests handling
Client/server WebSockets support
Transparent content compression/decompression (br, gzip, deflate, zstd)
Multipart streams
Static assets
SSL support using OpenSSL or Rustls
Middlewares (Logger, Session, CORS, etc)
Integrates with the awc HTTP client
Runs on stable Rust 1.72+

Documentation

Example

Dependencies:

[dependencies]
actix-web = "4"

Code:

use actix_web::{get, web, App, HttpServer, Responder};
#[get("/hello/{name}")]
async fn greet(name: web::Path<String>) -> impl Responder {
    format!("Hello {name}!")
}

#[actix_web::main

…

View on GitHub

Last but not least we want to take a look at actix-web. It combines aspects from both previous frameworks, rocket as well as axum, which is resembled in its ergonomics. Annotations are back on the menu and macros are more heavily used, especially in the definition of route handlers. When implementing middleware or database connections, however, you will not find the same level of abstraction as you might with rocket. Actix is extremely fast 3 and it aims to cater to both experienced Rust developers as well as newcomers who are just starting with Rust development. It ships with its own middleware, e.g. for logging, session management, or cross-origin resource sharing, and allows you to expand the framework with your own middleware that can hook into actix. Let’s have a look at how route handlers, database connections, and starting the server are handled in actix.

#[post("/texts")]
async fn post_text(client: web::Data<Client>, payload: web::Json<TextResponse>) -> impl Responder {
    // ...
}

#[get("/texts/{uuid}")]
async fn get_text(client: web::Data<Client>, uuid: web::Path<Uuid>) -> impl Responder {
    // ...
}

Again, we have a lot less boilerplate. Though launching the server falls in between the first two examples with regards to verbosity.

#[actix_web::main] // or #[tokio::main]
async fn main() -> std::io::Result<()> {
    let db_client = Client::with_uri_str("mongodb://....").await.expect("failed to connect");

    HttpServer::new(move || {
        App::new()
            .wrap(Logger::default())
            .app_data(web::Data::new(db_client.clone()))
            .service(post_text)
            .service(get_text)
    })
    .bind(("0.0.0.0", 8080))?
    .run()
    .await
}

The benchmarks speak for themselves, actix is indeed blazingly fast! And that does not come at the cost of developer experience, which actix manages to keep to a very high standard. It ships with more batteries included than axum, but does not abstract everything quite as much as rocket. If you already feel comfortable writing programs in Rust and other low-level web programming languages, you will enjoy developing in actix, and your user will enjoy the performance!

Acknolegements

Selim wrote the initial draft of this post and Selim, Zak, and I each implemented the sample API using one of the above frameworks.

Cover image: "Brown Chains" by Miguel Á. Padriñán

Selim KaelinFollow

I am a mountain athlete at heart, enjoy learning about and writing code, and passionate about Rust, Cloud, and maps!

Zak CookFollow

Jan KleineFollow

Two Years in the Vault: 4 Best Practices 🔒

Zak Cook — Mon, 09 Dec 2024 08:30:00 +0000

I work as an IT consultant. Over the past two years, we've been working with our client on a platform engineering project, where the use of HashiCorp Vault for secrets management was prevalent. Our Vault has enabled us to manage hundreds of credentials, increasing the security of our developer platform, and has resulted in a thorough and battle-tested configuration of our Vault which we can be proud of.

However, over the past two years, there have been quite a few learning moments. Vault is a great product with lots of flexibility, but this opens up a lot of possibilities for misconfiguration or misuse. For us, that meant a few tedious reconfigurations of our Vault. This post aims to share learnings that I wish we had known from the very beginning, in the form into digestible tips.

These tips are for everyone that needs to work with HashiCorp Vault, whether it be as a developer, or as an administrator (you may still learn something). If you don't use Vault, maybe simply bookmark this post, it may come in handy in the future.

TL;DR:

Configure with an infrastructure-as-code (IaC) tool
Policies describe permissions
KVv2: save raw data, format elsewhere
Beware of write permissions on sys/policy
Consider templated policies (upcoming)
KVv2: make secrets "connection bundles" (upcoming)
Utilize -output-policy from the CLI (upcoming)
Think about your path structure (upcoming)

This is a two-part series, I'll cover the first four tips here, and the remaining four in an upcoming post. Stay tuned.

Glossary: I overuse the words component and system a lot. This could refer to microservices, short running jobs, monolith servers, which all run together, to form some sort of workload. If you're running on Kubernetes, think of a component as a Deployment, and a system as the Kubernetes cluster.

Tip 1: Configure with an infrastructure-as-code (IaC) tool

When first starting to integrate a Vault into a large system, one naturally does explorative work locally with the Vault CLI. This is all good, but concretizing that configuration into infrastructure-as-code (IaC) is essential, and by doing it early you'd avoid the pain of migrating your configuration down the road.

Potential solutions for IaC are:

For configuring your Vault, I would generally recommend going for Terraform/OpenTofu, which takes the declarative approach as opposed to Pulumi. Pulumi is a good solution when you have lots of dependencies in a complex system, which need to be handled programmatically.

We started off on our Vault journey by simply documenting how we configured our secrets engines, auth methods and policies. So when we had to set up a new Vault, that meant going through that documentation and running the commands one-by-one. We at some point (painfully) changed to Terraform/OpenTofu, and it directly opened up many doors for us:

Writing a testing framework, where we would test the access to Vault paths by simulating the components in our system, allowing us to check for any regressions
Reusing the IaC to configure Vaults in multiple environments
Integrate with CI pipelines to automate updates to our Vaults

There's many more advantages. Point is, start with IaC directly, so you don't have to go through the same process as we did, of integrating an already configured Vault with IaC. In case you have to: Terraform Import Blocks saved our lives :)

Tip 2: Policies describe permissions

Policies are core to HashiCorp Vault, being one of the pillars in its access control mechanisms. A Vault policy is a set of permissions, each being a combination of a path and capabilities on that path. Efficient management of policies is important in order to keep your Vault lean and scalable.

Let's consider a very basic Vault, which has a KVv2 engine under secrets/, used to store secret recipes under the recipes/ subpath. The following policy could be the "head chef" policy, allowing the head chef to browse the entire secrets engine and update any of the secret recipes:

vault policy write head-chef - <<EOF
# browse secrets
path "secrets/metadata/*" {
  capabilities = ["read", "list"]
}
# manage recipes
path "secrets/data/recipes/*" {
  capabilities = ["create", "update", "read", "delete"]
}
EOF

We would then attach this policy to a role with:

vault write auth/approle/role/head-chef token_policies=head-chef

Here we decided that the name of the policy would match the role, namely head-chef. This works, but we have found that as policies become larger and more roles are added to the Vault, breaking down policies into smaller, more meaningful sets of permissions becomes increasingly important. Policy names should also describe the permissions encapsulated, and should not refer back to the role that uses it.

To help breaking down policies, it is useful to decide on a naming standard for your policies. An example would be <verb>-<subject of policy>, like read-nuclear-codes. In this example we would have to break down the policy a bit more in order for our convention to apply. Let's split the head-chef policy into two policies, browse-secrets and manage-recipes:

vault policy write browse-secrets - <<EOF
# browse secrets
path "secrets/metadata/*" {
  capabilities = ["read", "list"]
}
EOF
vault policy write manage-recipes - <<EOF
# manage recipes
path "secrets/data/recipes/*" {
  capabilities = ["create", "update", "read", "delete"]
}
EOF
vault write auth/approle/role/head-chef token_policies=browse-secrets,manage-recipes

It is now much more transparent what permissions are associated with this role. In addition, breaking down the policy made parts of the policy reusable, as we could now do something like the following, reusing the browse-secrets policy:

vault policy write manage-greek-recipes - <<EOF
# manage greek recipes
path "secrets/data/recipes/greek/*" {
  capabilities = ["create", "update", "read", "delete"]
}
EOF
vault write auth/approle/role/chef-george token_policies=browse-secrets,manage-greek-recipes

Deciding on this naming convention for our policies allowed us to scale to many more roles in our Vault. One thing to consider though, is that paths may end up overlapping among the policies assigned to a role. This requires being aware of how Vault's priority matching works.

Tip 3: KVv2: save raw data, format elsewhere

Vault's KVv2 engine allows for saving static secrets in the form of key-value pairs. Choosing the appropriate key-value pairs to save in a secret will both maximize flexibility in how your infrastructure consumes these secrets and would minimize duplicated data.

Let's say you have a component which requires credentials to a database, and needs them provided inside a config, e.g. in TOML format. Since the config holds secrets, it may be tempting to save the entire config in Vault's KVv2 engine, and fetch that config when starting your component:

vault kv put secrets/components/my-component config.toml=- <<EOF
[database]
host = "database.example.com"
username = "postgres"
password = "1234"
EOF

However, this approach is limiting:

Reusing the database credentials means copy-pasting to another config, and doesn't allow for a source of truth
Any non-secret config changes need to go over the Vault

This can be avoided: the Vault ecosystem has a large amount of helper components which support templating, among which:

Vault agent templating
External Secrets Operator templating in case you're running on Kubernetes.

As an example let's make use of an ExternalSecret for our templating. First, let's save the standalone credentials instead of our config file:

vault kv put secrets/databases/my-database username=postgres password=1234

Now we'll write an ExternalSecret making use of this raw data:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: config
spec:
  # ... (reference to SecretStore)
  target:
    template:
      engineVersion: v2
      data:
        config.toml: |
          [database]
          host = "database.example.com"
          username = {{ .username | quote }}
          password = {{ .password | quote }}
  dataFrom:
  - extract:
      key: /secrets/databases/my-database

This would then render a Secret resource with the following content under the config.toml key, which can be mounted in our component:

[database]
host = "database.example.com"
username = "postgres"
password = "1234"

The structure of our KVv2 engine is now cleaner, as it is not aware of the components that access it, and holds sensible reusable secrets. Another option is to tweak how the component is configured. Instead of taking credentials directly, our component could take a Vault path instead:

[database]
host = "database.example.com"
vault_path = "secrets/databases/my-database"

This however means a dependency on HashiCorp Vault in the code of the application itself, as the code now uses the Vault API in order to retrieve the secret. Some applications may want to be agnostic to the type of software storing the credentials, in which case a templating solution would be more suitable.

Tip 4: Beware of write permissions on `sys/policy`

Understanding this tip is a bit more involved, but please bear with me. It is arguably the most important tip in this series, especially in environments where security is critical.

Vaults used in a complex system sometimes need to support new users being asynchronously added to the Vault: for example adding a new role/user which has access to a subdirectory of a KVv2 secrets engine. To automate this, companies may resort to building some management component which automates the creation of a role/user and policies attached to it.

The sys/policy (or sys/policies/acl) path is used to grant permissions on managing policies in Vault, and is necessary to create policies on the fly. We will see in this chapter how granting write permissions on sys/policy can lead to severe security risks, and provide some examples on how to mitigate these risks.

As an example, let's consider our Vault which is meant to manage recipes. Our company wants to support employees requesting their own subdirectory in our secrets/ KVv2 engine. This would involve the employees sending an API request to a component, let's call it the employee-manager, that would need to execute following commands on the Vault upon receiving one of these requests:

# create policy for employee
vault policy write "manage-recipes-$EMPLOYEE_NAME" - <<EOF
path "secrets/data/recipes/$EMPLOYEE_NAME/*" {
  capabilities = ["create", "read", "update", "delete"]
}
EOF
# create role for employee with attached policy
vault write auth/approle/role/$EMPLOYEE_NAME token_policies="manage-recipes-$EMPLOYEE_NAME"

This would require the employee-manager component to have the necessary permissions to create both roles and policies for our employees. Let's setup a role and policy to allow the component to execute the above commands:

# create "create-policies", "create-roles" policies
vault policy write "create-policies" - <<EOF
path "sys/policy/+" {
  capabilities = ["create", "update"]
}
EOF
vault policy write "create-roles" - <<EOF
path "auth/approle/role/+" {
  capabilities = ["create", "update"]
}
EOF
# attach policies to role
vault write auth/approle/role/employee-manager token_policies=create-policies,create-roles

The above permissions are extremely dangerous to provide to any component or user of Vault, especially in conjunction.

Imagine our employee-manager component was insecure, and is compromised by an attacker. The attacker then uses the component's credentials to log in to Vault. They can now create new policies, with any permissions, and assign those permissions to the employee-manager which they have control over:

# create malicious policy
vault policy write "read-secrets" - <<EOF
path "secrets/data/*" {
  capabilities = ["read"]
}
EOF
# update the compromised role with malicious policy
vault write auth/approle/role/employee-manager token_policies=create-policies,create-roles,read-secrets

# after logging in again, the attacker could execute:
vault read secrets/data/recipes/gordon/super-secret-meatballs-recipe

So the component must technically be considered as admin on the Vault, as it can create policies for absolutely anything and assign it to itself. To mitigate this security risk, there are the following solutions:

Avoid granting permissions on sys/policy altogether and consider using templated policies. This is often the cleanest solution, and has a big advantage of reducing the amount of configuration.
Grant permissions on a subfolder of sys/policy. This subfolder must be disjoint from the policies granted to employee-manager, and employee-manager should not be able to update its own role.
Grant only create permissions on anything under sys/policy, and employee-manager should not be able to update its own role.

There may be more mitigation options, but the point of this chapter is to consider the event of a breach and be aware of the effective permissions that you may be giving to your system components.

That's all the tips for now. Thanks for reading, I hope you learned something, and stay tuned for the second half and other posts to come.

DEV Community: Innovation Process Technology AG (ipt)

Generative UI - The Future of Human-AI Interaction

Benjamin BürgisserFollow

TL;DR

Why Generative UI?

The Three Core Patterns of Generative UI

Key Generative UI Frameworks

Introducing CopilotKit

Our Demos

Demo 1: Tic-Tac-Toe Assistant

Demo 2: 2D RPG with LangGraph NPC

A Quest in Three Acts

Lessons Learned

What Worked Well

What Tripped Us Up

Is Using CopilotKit Worth It?

Conclusion

From Rot to Rhythm: How We Rebuilt Our Documentation Culture with Docusaurus

The Forgotten Layer

2. The Slow Decay

3. The Moment Everything Snapped

4. Why Confluence Couldn't Keep Up

5. Before the Tool Came the Story

6. Choosing the Right Documentation Platform

7. Understanding Docusaurus (for those new to it)

8. Our Docusaurus Documentation Platform

What our Docusaurus setup provides

Sharing is Caring!

9. The First Migration

10. Documentation as an Enabled Ecosystem

11. What Surprised Us Most

12. Lessons Learned Along the Way

13. Why We're Sharing Our Scaffold

14. The Moral of the Story

Beyond the Pod: Why wasmCloud and WebAssembly Might Be the Next Evolution of the Platform

WebAssembly as a Platform Foundation

WebAssembly: The Component Model

WebAssembly: Secure by Default

WebAssembly: Performance

wasmCloud Architecture

Hosts

Lattice

Capabilities

Providers

Components

Applications

wadm

Verdict

That Time We Found a Service Account Token in my Log Files

What This Article Covers

Why This Matters

What Is a JWT?

Closer Look at aud

Kubernetes and ServiceAccounts

ServiceAccounts 101

The Catch

Enter Vault: The Gatekeeper of Secrets

Vault Authentication Methods

Kubernetes Auth Method

JWT Auth Method

Projected Tokens: Because It's 2025

What You Get

Example Pod with Projected Token

Why Vault Loves This

Back to the Log

Try It Yourself: Vault + K8s AuthN Lab

What's Inside

Final Drop 🎤

Acknowledgment

Dissecting Kubewarden: Internals, How It's Built, and Its Place Among Policy Engines

Policy Engines

Kubewarden Architecture

Comparison

Verdict

Your Cluster Deserves Better Traffic Management. Enter: Gateway API.

Current Alternatives

NodePort: Exposing a Service on a Node’s Port

LoadBalancer: External Load Balancers for Services

Ingress/Routes: Layer-7 Routing for HTTP/S

From Ingress to Gateway API: Why a New API?

Closer Look at `aud`

A command execution engine powered by `rofi` and `fzf`.