DEV Community: Jakob Beckmann

Beyond the Pod: Why wasmCloud and WebAssembly Might Be the Next Evolution of the Platform

Jakob Beckmann — Tue, 21 Oct 2025 05:08:44 +0000

Over the past few months I have invested some time to contribute to an open source project I find fascinating: wasmCloud. As a platform engineer and architect, I am very familiar with how software platforms are typically built in practice. However, with the ubiquity of Kubernetes, you run the risk to being stuck in the "doing it the Kubernetes way" line of thinking. But then again, are there any better ways? This is where wasmCloud caught my attention. A modern platform building on proven concepts from Kubernetes, but with some significant differences. In this article I want to introduce wasmCloud, how it compares to Kubernetes, what its internal architecture looks like, and what ideas are, in my humble opinion, a step up from "the Kubernetes way of things".

Before getting started, I need to get some things out of the way. This article will make quite a few comparisons to Kubernetes and bytecode interpreters like the JVM. If you are unfamiliar with these technologies, it might make sense to have a short look at what these are. Considering you clicked on this article, I am however guessing that you are familiar with them and have some experience in platform engineering practices, either as a poweruser of a platform, or as a designer and developer of one.

Moreover, I want to thank the company I work for, ipt, for allowing me to invest time to learn about new technologies such as wasmCloud. Not only is contributing to open source a great way to pay back a community powering the modern world, it is also a huge passion of mine. Being able to help the development of such projects during paid worktime enables me to learn so much on emerging technologies, and maybe help build the revolutionary tools of tomorrow.

So... wasmCloud!? I have been interested in WebAssembly ever since it promised to replace JavaScript, a language I personally consider as extremely poorly designed (someone once told me it was designed in three days, so no wonder there). While WebAssembly is very far from doing anything close to replacing JavaScript in the browser, it has evolved into something else: an application runtime and a potential replacement for containers.

WebAssembly as a Platform Foundation

Modern platforms nearly all build on top of containers as their foundational element to run executable code. This is a logical evolution from Docker's meteoric growth, and the ecosystem that grew around its open standards (such as the OCI - Open Container Initiative). While containers provide a huge step in terms of ease of use, standardization, and security compared to shipping raw artefacts to virtual machines, as was the case before them, they do have some shortcomings.

First and foremost, containers are not composable. In part due to their flexibility, they do not offer standard ways of expressing how the world should interact with them at runtime, or what they rely on to perform their functionality. This means that containers are typically deployed as REST-based microservices, where containers communicate with one another over a network using APIs agreed upon outside of the container standards. This lack of standardization makes building reusable components more challenging than it has to be. Moreover, each container essentially needs a server, authentication, authorization, and more to run. This results in quite some waste in the compute density of the platform, with lots of compute wasted on boilerplate.

Moreover, while containers are a huge step in the right direction in terms of security, they are not quite as secure as most people are led to believe. Containers are "allow by default" constructs, which take quite some work to properly harden.

Finally, due to how containers are typically built, their startup times are not that great. It is not abnormal to see container start times in the dozens of seconds. This does not bother people very much because containers are mostly used to run long running processes (since we need these REST APIs everywhere). However, a large part of containers are mostly idle, waiting for some API request to come in. If one considers that workloads could be called (and thus the process started) only when needed, startup times over 100ms is considered slow.

This is where WebAssembly comes it. WebAssembly addresses these challenges. Composability is addressed by the component model.

WebAssembly: The Component Model

The component model is a way that WebAssembly modules can be built with metadata attached to them which describe their imports and exports based on a rich type system. Moreover, they are composable such that a new component can be built from existing components as long as the imports of one are satisfied by the exports of another. This means that components can interact with one another via direct method/function calls, whose specification is fully standardized. This interface specification is declared in a language known as the WebAssembly Interface Types (WIT) language. An example of a WIT specification of a component relying on a system clock can be seen below:

package wasi-example:clocks;

world mycomponent {
    import wall-clock;
}

interface wall-clock {
    record datetime {
        seconds: u64,
        nanoseconds: u32,
    }

    now: func() -> datetime;

    resolution: func() -> datetime;
}

WIT can be compared to the Interface Definition Language (IDL) from gRPC but for wasm components.

This declaration says that the component relies on an interface wall-clock (it imports the interface) which defines two functions: now and resolution. Both take no arguments and return a datetime object consisting of a seconds and nanoseconds field. This component could then be composed with any other component which exports this wall-clock interface.

If this were a container which would rely on accessing some API, we would need to read a non-standardized documentation of the container image, and then read up on other containers to ensure they provide APIs that match the ones called by the first container.

The WebAssembly component model can essentially be seen as a form of contract-based programming to formalize interfaces between WebAssembly core modules.

WebAssembly: Secure by Default

Whereas containers provide some form of security by namespacing processes and filesystems, WebAssembly actually sandboxes modules such that they cannot affect one another, or the host they run on. By default a WebAssembly module cannot perform any privileged action and needs to be granted explicit permission. I will not dive deeper into the details of this or I might loose myself in a rant on how software security in the modern day and age is abysmal.

WebAssembly: Performance

WebAssembly's main goal is performance. This means that WebAssembly modules run fast, but also that loading modules and starting them is much faster than containers. This has proven to be very useful already, for instance in use cases such as serverless computing, where hyperscalers heavily rely on WebAssembly as a runtime to reduce cold start times, and reduce the delay in function calls.

Considering the idea to avoid having long running servers providing REST APIs and move to raw function calls on short running modules, having extremely short start times is imperative.

Alright, so we can see that WebAssembly can be a great choice for the foundation runtime of a platform. So where are platforms leveraging this? Well, actually, quite some "platforms" leverage this idea already. For instance, SpinKube does exactly this, enabling to run WebAssembly functions on Kubernetes. However, you still interact with these functions via a REST call. Another example is Kubewarden, leveraging WebAssembly modules to evaluate policies. While some might argue that this is not a platform, Kubewarden provides a runtime for arbitrary programs, including their scheduling and deployment. Sounds like a platform to me.

Finally: wasmCloud! wasmCloud is probably what people would consider the closest to a full blown platform to run WebAssembly modules. In other words, what Kubernetes is to containers, wasmCloud is to WebAssembly components. It provides a way to deploy, schedule, link, and lifecycle WebAssembly components on a distributed platform.

wasmCloud Architecture

Let us look at the wasmCloud architecture a little.

This section will contain quite a few comparisons to Kubernetes concepts.

Generally, the wasmCloud architecture can be seen as quite similar to the Kubernetes architecture, with the difference being that wasmCloud does not provide as much flexibility in swapping out building blocks as Kubernetes does. This makes sense as it is a more nascent technology and is currently more opinionated.

As a reference, here is the diagram wasmCloud uses to provide an overview of the platform:

As one can see, the architecture is essentially a set of hosts connected via a so called "lattice". Thus, the architecture distributes the runtime over a set of compute instances in order to achieve resilience against hardware/compute failures. The principle is identical to the one from Kubernetes, providing a cluster in order to be able to quickly shift payloads on the platform to different nodes in case of node failures.

Hosts

wasmCloud hosts are the foundation of the compute platform that is provided. They are the equivalent of Kubernetes nodes and provide a WebAssembly runtime for components to run on. Just as with Kubernetes nodes, application developers will rarely need to worry about the hosts other than for deployment affinities and the like.

In practice, hosts can be anything from a virtual machine, an IoT device, or even a pod running on Kubernetes. In fact, hosting wasmCloud on Kubernetes is a relatively straight forward way to get started with the technology, providing wasmCloud as an application runtime, while providing services via Kubernetes.

Lattice

The wasmCloud lattice is its networking layer. This can seem a bit strange when considering that this a NATS instance.

For those unfamiliar with NATS: it is an event streaming component similar to Kafka, but provides additional features such as a key values store, an object store, and publish-subscribe capabilities.

Having a NATS instance as the "networking layer" confused me quite a lot at first. However, one has to remember that thanks to the component model, we no longer require HTTP/TCP network calls for our components to interact with one another. Thus we don't necessarily need an IP to address a component we want to reach. Of course NATS itself will require a physical network to run on in order to distribute events to its different instances, but wasmCloud then only needs to use NATS.

Essentially, every component exposing a function becomes a subscriber to a queue for this function on NATS. Other components can then call this function via wRPC (gRPC for WebAssembly) by publishing a call to some subject. This is quite different from Kubernetes networking, where calls need to know the location of the callee in the network. Using a subject-based addressing model simplifies deployment and improves scaling and resilience.

As a user of wasmCloud, you do not need to worry about this though. How function calls are preformed under the hood is abstracted away from the user.

This distributed networking aspect is one of the superpowers of wasmCloud, as one does not need to worry about how to address a component on the platform. However, it can also introduce strange behaviour in some cases. For instance, on Kubernetes, it's common sense that a HTTP call to a different pod running on the cluster can fail. On wasmCloud however, if the interface we are calling from a different component returns some type, we use the component like a raw function call in our components code. What if that call fails, not because of the called component but due to a networking issue? In the current implementation of wasmCloud this will lead to a panic in the caller. As this is typically not the desired outcome, efforts are underway to design an adapted way how the interfaces need to be designed to handle failures in the transport layer. On top of that, function calls might change such that might avoid using NATS as a transport layer if the component being called in on the same host and the caller.

Capabilities

This is where Kubernetes and wasmCloud start differing in their philosophy. Thanks to the standardized way interfaces can be declared in the component model, one can describe an abstract interface which provides some functionality, without providing an implementation. This is what capabilities are. They are abstract interfaces that describe some useful functionality, such as reading and writing to a key value store, or retrieving some sensitive information from a secured environment. These capabilities are published on wasmCloud for applications to use.

An application developer can then write a component that makes use of that interface if he/she needs that functionality. He/she does not need to worry about how this capability is implemented. He relies on the "contract" provided by the capability.

In my opinion, while this is quite challenging to grasp initially, this is what makes wasmCloud so promising. Having worked on many platforms in the past, the main challenge is always how additional services can be provided on top of raw platforms such as Kubernetes in a way that makes then highly standardized while easily consumable. In the current state of platform engineering, this quickly becomes a question of good product management. Unfortunately, doing this correctly is surprisingly difficult. Capabilities provide a technical solution to this, with the only limitation being complete incompatibility with existing software.

Providers

A provider is a specific implementation of a capability. For instance, taking the example of the capability enabling the reading and writing to a key value store, a provider might implement this by having a ValKey instance backing the capability. Another provider might implement the very same capability using NATS, Redis, or even an in-memory key-value store.

Abstracting the provider away from the consumer via a capability enables the platform to swap providers based on needs. Of course performing such a swap might be quite complex, for instance involving a data migration from NATS to ValKey. However, the beauty is that the applications do not require any changes as would be the case in traditional platforms.

It should be noted that the provider might run completely outside of wasmCloud itself. However, wasmCloud also provides internal providers that are backed into the hosts themselves, providing functionality such as logging or randomness.

Components

Components refer to the WebAssembly payload that contain your business logic. In the traditional sense, this is your application. However, in wasmCloud lingo, an application is a set of interlinked components including all information about what capabilities they require.

Applications

Applications are an abstraction enabling to declaratively define a combination of components, capabilities, and providers together into a deployable unit. Applications are based on the open application model (OAM) and should thus look quite familiar to people working with Kubernetes. In terms of definition, they are similar to a Kubernetes Deployment, describing not only the deployment unit (component or pod in the Kubernetes context), but also its replication, affinities, links to capabilities, etc.

It should be noted that in wasmCloud v2, applications are re-worked to be much more closely modelled after Kubernetes Deployments and ReplicaSets. Version 2 drops the idea of Applications alltogether and uses Workload, WorkloadReplicaSets, and WorkloadDeployments objects. These are also no longer linked to the OAM. In all likelihood we will write another blog post showcasing the capabilities of composition provided by version 2 in the future.

wadm

The wasmCloud Application Deployment Manager (wadm) manages Applications. It can be seen as the deployment controller from Kubernetes for wasmCloud Applications. It essentially orchestrates the deployment of components, capabilities, their links, etc. on the platform. This construct will also be dropped with wasmCloud version 2.

Verdict

With a decent understanding of the architecture we can now get an idea of the uses of wasmCloud in the real world. While I have not yet run anything productive on wasmCloud, I have played with the platform a lot over the past few months, and have come to really appreciate some of its innovative ideas.

Thus, to summarise my experience: wasmCloud is a relatively new platform and provides interesting new approaches to how inter-component communication can be modeled. On top of that, it does it while building on open standards such as WebAssembly and the component model, such that the business logic of your application remains portable. While these new concepts are very promising, wasmCloud still suffers from a couple drawbacks:

For people unfamiliar with WebAssembly, it has a quite steep learning curve. This is highly accentuated for people unfamiliar with existing platforms such as Kubernetes.
The set of supported providers and capabilities is extremely small to date. This will of course grow as adoption increases, but currently early adopters will have to write their own providers most of the time and will not be able to rely on third-party components.
As wasmCloud shifts more responsibility to the platform level, it will require a strong platform team to operate this with low developer friction. This can be an issue as finding highly skilled platform engineers is quite difficult at the moment. However, the team behind wasmCloud is focused on making application delivery as frictionless as possible.
Finally, I am not sure I currently understand the security model wasmCloud uses to authenticate and authorize calls between components. While I am not sure this is a drawback, it does not yet feel as intuitive as Kubernetes simple yet relatively powerful RBAC. I will have to dive deeper into this to have a final opinion on it though (another blog post might follow on this).

Dissecting Kubewarden: Internals, How It's Built, and Its Place Among Policy Engines

Jakob Beckmann — Mon, 07 Jul 2025 05:31:30 +0000

Kubernetes offers amazing capabilities to improve compute density compared to older runtimes such as virtual machines. However, in oder to leverage the capabilities of the platform, these tend to host applications from various tenants. This introduces a strong need for properly crafted controls and well-defined compliance to ensure the tenants use the platform correctly and do not affect one another. The RBAC capabilities provided out of the box by Kubernetes are quickly insufficient to address this need. This is where policy engines such as Kubewarden come into play. In this post we will look at how Kubewarden can be leveraged to ensure correct usage of a platform, how it compares to other policy engines, and how to best adopt it.

Policy Engines

Kubernetes provides role-based access control (RBAC) out of the box to control what actions can be performed against the Kubernetes API. Generally, RBAC works by assigning sets of roles to users or groups of users. Capabilities are attached to these roles, and users having a role obtain these capabilities. This simple mechanism is very powerful, mostly because it is quite flexible while allowing a simple overview of a user's capabilities. However, in the case of Kubernetes, the definition of capabilities is very restricted. Roles only allow or deny access to Kuberenetes API endpoints, but to not allow control based on payload content. This means that these capabilities are mostly restricted to CRUD operations on Kubernetes primitives (e.g. Deployments, Ingresses, or custom resources). Unfortunately, this is often not enough.

For instance, it is quite common to allow users to perform actions on some primitives under specific conditions. An example would be that creating Deploymentss is only allowed as long as its name follows some convention and the pods its creates are not privileged and set proper resource requests/limits. The naming convention cannot be enforced by standard RBAC controls as these have no possibility to represent more complex logic. Controlling the configuration of the pods created by a Deployment is a validation of the payload pushed to the API, and is thus not supported either.

Note: Security contexts and resources on pods can be controlled via methods such as Security Context Constraints or Pod Security Policies and ResourceQuotas. However, these do not reject the creation of the deployment, but will only block the creation of the pods themselves. It is therefore possible to apply a Deployment that is known to not allow the creation of pods. In my personal opinion this is not ideal, as it does not fail early.

These scenarios is where policy engines come into play. They utilise Kubernetes' Dynamic Access Control mechanisms to enable cluster administrators to manage permissions using more complex logic. The exact capabilities of policy engines can vary greatly as these are essentially arbitrary software that validates or mutates Kubernetes requests. However, the majority of major policy engines work similarly. They tend to implement the operator pattern, enabling the configuration of policies using Kubernetes custom resources. In this blog post we will have a look at Kubewarden in more detail, and how it compares to other engines.

Kubewarden Architecture

Kubewarden leverages WebAssembly (WASM) to enable extremely flexible policy evaluation. Essentially, Kubewarden can be seen as a WASM module orchestrator where policies are deployed as serverless functions that get called when necessary. The result of these WASM functions then determines whether an API request against Kubernetes is allowed, denied, or altered (mutated).

This similee can also help explain Kubewarden's architecture. Essentially, the Kubewarden controller (operator) manages policy servers and admission policies. Policy servers can be seen as hosts for the serverless execution of functions, whereas admissions policies are the functions themselves. Therefore, in order to perform policy validation, one needs at least one policy server running to host the policies one wants to enforce. The controller then takes care of configuring the runtime (policy server) to properly run the adequate policy executable with the appropriate inputs when a policy needs to be evaluated. The diagram below illustrates this:

As policies are WASM modules, they can themselves support configuration. This makes policy reuse a major feature of Kubewarden. Complex logic can be contained in the WASM module while exposing some tuning as configuration, allowing a policy to perform a relatively generic task. To understand this better, let us have a look at such a policy:

apiVersion: policies.kubewarden.io/v1
kind: ClusterAdmissionPolicy
metadata:
  annotations:
  name: "cel-policy-replica-example"
spec:
  module: registry://ghcr.io/kubewarden/policies/cel-policy:v1.0.0
  backgroundAudit: true
  mode: protect
  mutating: false
  policyServer: default
  rules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments"]
  settings:
    variables:
      - name: "replicas"
        expression: "object.spec.replicas"
      - name: maxreplicas
        expression: int(5)
    validations:
      - expression: "variables.replicas <= variables.maxreplicas"
        messageExpression: "'the number of replicas must be less than or equal to ' + string(variables.maxreplicas)"
  namespaceSelector:
    matchLabels:
      environment: test

In this example, we are using a WASM module which evaluates a Common Expression Language (CEL) expression to define our policy. Evaluating a CEL expression is not something we want to implement every time ourselves. Thankfully, Kubewarden provides this as a WASM module on their ArtefactHub. Thus we do not need to implement anything and can reuse that module. It is referenced on the module line above. Of course we also need to actually define the CEL expression that should be the heart of the policy rule. This is done within the settings block. Note how we can use object internals (such as replicas defined in a Deployment) in the validation expression. Finally, we need to define on what objects this policy should be evaluated. In order to do this, we provide rules that tell Kubewarden on what Kubernetes API endpoints to trigger the policy, and additionally provide information about which namespaces should be affected by the policy with a namespaceSelector. The remaining options configure the following:

backgroundAudit: informs Kubewarden to report on this policy for objects that are already deployed. In this case, we validate the replicas on created or updated Deployment objects. However, there might already be Deployments on the cluster that violate the policy before we start enforcing it. This option will tell Kubewarden to provide reports on such violations.
mode: Kubewarden supports enforcing policies (in protect mode), or monitoring the cluster (in monitor mode). Using the monitor mode can be interesting when investigating how people use the Kubernetes cluster or providing them with warnings before enforcing policies.
mutating: policies can also mutate (change) requests. In this case we are only performing validation to potentially reject requests. Thus we set mutating to false.
policyServer: as explained above, Kubewarden can manage many policy servers. This simply informs the controller on which policy server this specific policy should be deployed.

As one can see based on the sample policy above, while Kubewarden technically uses programs as policies, it is usually not necessary to write any code to use Kubewarden. This is thanks to its strong focus on module configuration and re-usability. The above CEL module alone already enables the configuration of a very wide range of policies. On top of that, other modules shared on ArtefactHub provide more specific validations or mutations that might incorporate more complex logic. If this is not enough, policy groups (a feature we will not cover in this post) can be utilised to combine other policies and express more complex logic as well. Finally, if one has very specific needs that cannot be addressed by any of the publicly shared modules, one can still fall back to writing code and building ones own module with fully arbitrary logic. How such policies can be written, in actual code, might follow in a separate blog post.

The above architecture of Kubewarden is what makes it stand apart from most other policy engines. Generally policy engines contain the logic fully in the controller, only exposing configuration via the custom resource. Since Kubewarden can essentially execute arbitrary WASM bytecode, it is not bound by the expressiveness of the custom resource declaration.

All this considered, is Kubewarden the best choice for a policy engine and should be used in all scenarios?

Comparison

There are many other policy engines out there, such as Kyverno, Gatekeeper, or Polaris. So why would you choose Kubewarden over any other?

As explained above, Kubewarden provides unprecedented flexibility, thanks to the way it evaluates its policies. This has the massive advantage that you will never reach a point that you have a policy that you would like to enforce but are restricted by the policy engine itself. However, it also has some drawbacks. The primary one being complexity. Writing WASM modules is not for the fainthearted, as WebAssembly is not yet incredibly mature, and most developers will not be familiar with it. The complexity issue can however be sidestepped as the vast majority of policies can be expressed using off-the-shelf WASM modules provided by Kubewarden.

Another aspect that often needs to be considered in enterprise contexts, is support. Kubewarden is an open source project that is loosely backed by SUSE (as it was originally developer for its Rancher offering). Thus enterprise support is only available via a SUSE Rancher Prime. Other tools such as Kyverno are not only more mature, but offer more flexible enterprise support (via Isovalent).

Finally, another aspect to consider is the featureset of a policy engines. Not all policy engines support mutating requests, and are thus much more restricted in their use. However, in this category Kubewarden offers all the features typically desired from policy engines. Some engines such as Kyverno support more features such as synchronizing Secret objects. While this can be useful, it is, in my humble opinion, not a feature for a policy engine.

Of course, there are also personal preference aspects to consider. As an example, Kubewarden and Kyverno handle policy exceptions very differently. Kubewarden has matchers that can be defined as part of the policy itself, which allow to exclude some resources from being validated. Kyverno on the other hand uses a separate CRD called PolicyException. Both have advantages and disadvantages.

Verdict

Kubewarden is a very interesting piece of software. Its internal architecture enables it to be incredibly flexible, at the cost of complexity. However, due to a smart concept of WebAssembly module re-use, that complexity is mostly under the hood, unless one wants or needs to dive deep. In my opinion, Kubewarden can be an absolutely great consideration when ones operates very large Kubernetes clusters what might have quite exceptional requirements. However, even in these cases, I would recommend starting very slow, and slowly building up to the complexity Kubewarden can hold in store.

If you do not operate a large Kubernetes fleet, or expect to have rather standard requirements in terms of how you want to restrict access to you cluster(s), you might be better off with more mature and simpler tools like Kyverno. Getting support for these tools is likely to also be much simpler.

A large part of the complexity of Kubewarden also comes from all that is required to even run this in an enterprise context. Unless you allow pulling WASM modules directly from the internet, you will also need a registry to host OCI packaged modules. On top of that, should you decide to write your own modules, you will need a process to do this, and build knowhow in that area. These are some of the aspects I hope to cover in a follow up post.

The Tortoise and the Hare: do AI Agents Really Help for Software Development?

Jakob Beckmann — Wed, 23 Apr 2025 05:37:56 +0000

Making my development workflow as fast as possible is a big passion of mine. From customizing my development setup to get the last inkling of efficiency out of it, to thinking how to manage notes and knowledge resources to access them as quickly as possible. With the sudden ubiquity of AI in development tools, I came to wonder how AI could help me write code faster. Being quite the skeptic when it comes to AI actually generating code for me (using tools such as Cursor or GitHub Copilot), I came to investigate AI agents which specialise in code reviews. In this blog post I will share my experience using such an agent on a real world case. I will explore where such agents shine and where they are severely lacking.

I am an AI Skeptic

Generally I am not fond of using AI to develop software. My background is mostly in systems software, where correctness of the software can be critical. This means that using tooling that is non-deterministic and might not produce adequate results makes me uneasy. Furthermore, even if AI were to produce amazing results, a developer relying on it could quickly lose understanding of the code. This results in skill atrophy and large risks if the AI reaches the limits of its capabilities. In other words, I am not keen on having any AI generating code for me on a large scale for anything more than a proof of concept or low risk project.

Nonetheless, one would be foolish to ignore AI's capabilities when it comes to developer tooling.

AI Support Agents

Thus starts my journey investigating AI agents that can support me in the software development lifecycle, but whose main use is not to generate code. Many such agents exist, mostly focusing on reviewing code. I am quite the fan of such a use case, as the AI essentially plays the role of another developer I might work with. It reviews my code, provides feedback, suggestions, and potentially even improvements. It however does this immediately after I have opened a pull request, rather than having to wait for days or weeks on a human review.

How is this different from using an AI that generates code you might ask? The main difference lies in the fact that I still have to think on how to solve the problem I am working on, and provide a base solution. This forces me to understand the issue at hand. Thus, I am much better prepared to accept or reject any suggestions from an AI than if the AI just generated a first solution for me. Moreover, people (myself included) tend to be slightly defensive about the code they write. Thus I will, in all likelihood, only accept AI generated code improvements if it offers a real improvement, rather than blindly incorporating them into the codebase.

All in all, it is extremely unlikely that I will lose understanding of the codebase or have my problem solving skills atrophy, but I can iterate on reviews much faster.

CodeRabbitAI

In order to gain first experiences with such an AI agent, I chose to try out CodeRabbitAI. This was not a thoroughly researched decision. The main reason I chose CodeRabbitAI is that I could try it out for free during 14 days and that it integrates well with GitHub. I am aware that performance between AI models varies greatly. However, CodeRabbitAI uses Claude under the hood, a model typically known to perform surprising well on programming tasks. I thus expect it to not perform significantly worse than any other state of the art model out there.

Starting Small

In my opinion, such agents need to be tested on real world examples. One can see demos using AI to generate a dummy web app all over the place. However, common software projects are significantly larger, contain more complex logic, and are less standardized than these demos. Unfortunately, most software I work on professionally is not publicly available, so I cannot use CodeRabbitAI on these. I therefore picked two (still very small) personal projects of mine:

A NeoVim plugin providing a colour scheme.
A command execution engine to run templated commands.

Both projects are extremely small, with under two thousand lines of code. Both projects are written in Lua, a quite uncommon language. I wanted to see how the AI fares against something it is unlikely to have seen too much during its training.

With that in mind, I wrote a first pull request implementing a fix in highlight groups for pop-up menus in NeoVim. I enabled CodeRabbitAI to summarize the PR for me.

The summary looks good, even though it somehow marks some fixes as features. This is especially intriguing as I use conventional commits and explicitly marked these changes as fixes. Additionally, CodeRabbitAI offers a "walkthrough" of the changes made in the PR. In the case of such a simple PR, I found the walkthrough to be mostly confusing. In the case of larger PRs I can however see how this may be appealing.

In reality, I initially opened the PR with only the fixes the pop-up menus. I then pushed commits introducing the support for additional plugins later on. I would have expected CodeRabbitAI to complain that the new commits introduce changes unrelated to the PR, which is not seen as best practice. It did nothing of the sort.

While the summary, walkthrough, and disregard for best practices were unsatisfying, one unexpected benefit emerged: the integration of linting feedback directly within the pull request comments. It provided nitpicks from linting tools (in this case markdownlint. On one side, it is very disappointing to see that the AI agent did nothing more than lint the code and generate a nice comment out of the output. On the other hand it is quite nice that it introduces "quality gates" such as linting without me having to write a pipeline for it. Moreover, producing easily digestible output from a linter is nothing to be underestimated. The quality of life of having this directly as a comment rather than having to go through pipeline logs to read the raw linter output is quite nice. Is it worth two dozen USD per month? No, definitely not!

On the upside, it did update the summary of the PR to reflect the other changes:

The first PR was extremely trivial. It did not introduce any code containing logic. Other than not pointing out that it should probably have been two separate PRs, CodeRabbitAI fared as I would have expected another developer to have reviewed the PR. With two small differences:

The CodeRabbitAI review was close to immediate (took around 30-60 seconds to run). This is amazing to iterate quickly.
Where I would have expected a human reviewer to point our the nitpick or simply approve, CodeRabbitAI is extremely verbose with explanations, walkthroughs, and so on. This in turn wastes time for the author, as he/she would need to read through this. The verbosity could be nicer on larger PRs, but for small concise PRs this is massive overkill and borderline annoying.

To further evaluate CodeRabbitAI's capabilities, I decided to test it on a pull request with more substantial changes...

A More Complex PR

Armed with dampened expectations from my first PR, I opened another PR in the command execution repository implementing a feature affecting multiple files. These changes also update existing logic.

In this second PR, CodeRabbitAI went above and beyond, and generated a walkthrough containing two sequence diagrams showcasing the control flow of the code that was modified! I was actually quite impressed by this. While probably not necessary for the author of a PR, this is great even only for documentation purposes. New team members with less experience may benefit from such visual aids to understand complex logic within the code. Unfortunately the diagrams didn't highlight the specific modifications introduced by the pull request.

However, the supporting text suddenly becomes more relevant when considering such PRs.

On top of that, CodeRabbitAI actually posted interesting comments. It found the odd nitpick here and there, but also found more meaningful potential issues. For instance, I modified a test configuration to use a different shell. CodeRabbitAI identified that this shell is not listed as a dependency anywhere in the repository, and that it would thus not work off-the-shelf. In this case this was only a test file used to parse the configuration and the configured shell did not affect anything, but this is a great finding generally.

I also started conversing with CodeRabbitAI about some changes. Requesting it to give me a suggestion on some configurations. It managed just fine, but did not actually provide these as code suggestions that can be applied, but rather as code blocks in comments, which was a bit disappointing.

Additionally, I decided to try to use CodeRabbitAI's commands feature. This enables ChatOps to control actions taken by CodeRabbitAI. I generated the PR title using one such command. The title turned out generic and not very informative. In CodeRabbitAI's defense, I am quite unsure how I would have named that PR.

I then tried to get it to write docstrings for new functions that were introduced in the PR. It massively misunderstood the request, and created a PR adding docstrings to all functions in the affected files, even ones that already had docstrings... This goes to show that in some cases, it cannot even do what the most junior of all engineers would be capable of doing thanks to a even so tiny dose of common sense. Moreover, it started adding commits with emojis in the title. This goes to show that these AIs are probably not trained much on professional projects.

After that first disaster, with significantly less ambition, I requested it creates a PR to change a small typo. CodeRabbitAI informed me that it created a branch with the changes included, but that it was not capable of creating pull requests. This shocked me, considering it had created its first disaster PR no 10 minutes before.

After another nudge, CodeRabbitAI however did create a PR. It targeted main instead of the branch I was initially using. I guess this is my own fault though for not being specific enough.

Finally, I also tried to get it to update the wording on a commit it did to use conventional commits. Unfortunately it seems that it only has access to the GitHub API and cannot execute any local git commands. It is therefore not able to perform some relatively common operations in the SDLC that are not part of the GitHub API. However, I am guessing this is subject to change relatively soon with the emergence of technologies such as the model context protocol, which would enable it to control external tools such as git.

All in all, I would say CodeRabbitAI did as I would have expected after the first PR. It corrected nitpicks and allowed me to perform some simple actions. Did it deliver a review of the same quality like a senior engineer familiar with the project would have? No. In fact, in order to test this I intentionally implemented a feature that was already present in the repository, while making a couple design decisions that go against most of what the rest of the repository does. CodeRabbitAI neither detected that the logic I was introducing was already present in the codebase, nor did it complain about the sub-optimal design decisions. This goes to show that such agents are still not capable replacing humans with nuanced understanding of the project's history and architectural principles, potentially leading to the introduction of redundant or suboptimal solutions.

Dashboards!

Another feature of AI agents next to the reviews is the analytics capabilities that come with them. In my personal opinion, analytics are important to measure the impact the introduction of such tooling has on the software delivery. CodeRabbitAI provides a couple nice dashboards on how much it is being used, and what kind of errors it helped uncover.

I did not try out CodeRabbitAI for long enough to have any meaningful metrics, but I am confident that the capabilities provided are enough to get a decent understanding of the quality of adoption.

Moreover, CodeRabbitAI supports reporting. This allows to generate reports based on natural language prompts that could be useful for product owners to get insights of changes made to the software over the course of a sprint.

Verdict

While this whole article might seem like a slight rant against such tools, I would in fact wish I could use such tools at work. Not as a replacement for human reviewers, but as an addition to them. For instance, the quite verbose walkthroughs CodeRabbitAI provides can be a very helpful entrypoint to a human reviewer on larger PRs. Moreover, while the quality of the review is insufficient for projects where quality matters, having near instant feedback is amazing.

Finally, as mentioned above, I believe one major selling point of such agents is in the way we humans interact with them. Even if the agent might do little more than execute linters or similar in the background, having the output of these tools in natural language directly as comments in the PRs is not to be underestimated. This is especially true in the age where more and more responsibility is being shifted to developers. With DevSecOps, developers have to understand and act upon the output of all kinds of tools. Presenting this output in a more understandable format, potentially enriched with explanations, can have a significant impact.

Therefore, as a final word, I would actually encourage people to explore such agents to augment their workflow safely, albeit with caution and a clear understanding of their limitations.

f4z3r / gruvbox-material.nvim

Material Gruvbox colorscheme for Neovim written in Lua

Gruvbox Material

A NeoVim colour scheme in pure Lua allowing for highly flexible configuration and customization.

Features | Installation | Usage and Configuration | API Reference

Note

This is a continuation of the original work from WittyJudge https://github.com/WIttyJudge/gruvbox-material.nvim

A port of gruvbox-material colorscheme for Neovim written in Lua. It does not aim to be 100% compatible with the mentioned repository, but rather focuses on keeping the existing scheme stable and to support popular plugins. This colorscheme supports both dark and light themes, based on configured background, and harder or softer contrasts.

Dark theme:

Light theme:

Different contrasts

Contrast	Dark	Light
Hard
Medium
Soft

Features

Supported Plugins:

Please feel free to open an issue if you want some features or other plugins to be included.

…

View on GitHub

f4z3r / sofa

A command execution engine powered by rofi.

Sofa

A command execution engine powered by `rofi` and `fzf`.

About

sofa is a small utility to enable easy execution of templated commands. It can be used to store snippets that you often rely on, or fully template complex commands. It is meant to be used with a shortcut manager to enable launching from anywhere, but can also inject commands into your current shell session for commands that make more sense to run there (see Integration).

Examples

For Snippets Management

You can use sofa for standard snippets management. Use the integration described below, and have configuration such as:

Configuration

namespaces
  lua:
    commands:
      install-local:
        command: luarocks --local make --deps-mode {{ deps_mode }} {{ rockspec }}
        description: Install rock locally
        tags:
        - local
        - luarocks
        interactive: true

…

View on GitHub

A Comprehensive Guide to Managing Large Scale Infrastructure with GitOps

Jakob Beckmann — Tue, 08 Apr 2025 04:31:02 +0000

GitOps is getting adopted more and more. However, there still seems to be some confusion as to what GitOps is, how it differs from regular CI/CD pipelines, and how to best adopt it. In this post we
will quickly cover what GitOps is, and the three main lessons learned from using GitOps to manage infrastructure at scale both on premise and in the cloud.

GitOps Overview

GitOps is a set of principles enabling the operation of a system via version controlled, declarative configuration. More specifically, the OpenGitOps project defines four principles which define whether a system or set of systems is managed via GitOps:

Declarative: A system managed by GitOps must have its desired state expressed declaratively.
Versioned and Immutable: Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.
Pulled Automatically: Software agents automatically pull the desired state declarations from the source.
Continuously Reconciled: Software agents continuously observe actual system state and attempt to apply the desired state.

Note that git is not referenced anywhere, as GitOps is not bound to any tooling. However, in layman terms, many consider a system operated via git to be a GitOps system. This is not quite correct.

GitOps is More than CI/CD Pipelines

Taking the "layman's definition" from above, any system that has CI/CD via pipelines triggered on repository changes would be a GitOps system. This is not accurate. Consider an IaC pipeline which applies declaratively defined infrastructure (such as a standard opentofu apply in a pipeline, or a Docker build followed by a kubectl apply). While such a system adheres to the first two principles, it does not adhere to the latter two. This implies that changes made to the target system are not corrected (reconciled) until the pipeline runs the next time. Similarly, if the pipeline fails for whatever reason, the desired state does not change the pipeline: a configuration drift is not detected, even if not reconciled.

This is an important distinction when considering "standard CI/CD" and GitOps. Simply having something declared as code does not make it GitOps.

The Advantages of GitOps

GitOps has many advantages over standard ways of managing systems. The advantages of having a declarative desired state, version controlling it, and interacting with the system only via git (or whatever version control system you use) are tremendous. From improved security and higher efficiency to better change visibility. These are well known to most people and will thus not be covered here.

Drift detection and automatic reconciliation are the two other aspects that make GitOps absolutely amazing. This is especially true in the current day and age, with the proliferation of complex systems being worked on by many people concurrently. Being able to observe that the system is not in the desired state has massive advantages, such as for standard SRE operations. Continuous reconciliation ensures that manual operational tasks are kept to a minimum, and that systems cannot degrade over time as small undesired changes creep in.

Tooling

In this post we will mostly focus on using GitOps to manage resources handled via the Kubernetes API, but it should be noted that GitOps as a concept is in no way restricted to Kubernetes. In the Kubernetes space there are two major players for GitOps: ArgoCD and FluxCD. We will not go into the details as to what the advantages for each tool are, other than saying that according to our own experience, ArgoCD might be more developer focused, while FluxCD might suit platform engineers with more Kubernetes experience that want more flexibility.

The rest of this post is tool agnostic and everything we are talking about can be done with either tool (but some aspects might be easier to do with one or the other).

Infrastructure: Disambiguation

Before we dive into how to structure your GitOps configuration, it might make sense to draw a line as to where infrastructure starts and where it ends. We consider infrastructure everything that is part of the platform provided to an application team. Hence this line might vary depending on the maturity of the platform you provide your teams. If we consider a simple Kubernetes platform with little additional abstraction for its users, the infrastructure would contain the Kubernetes platform itself as well as all system components that are shared between the teams, such as a central monitoring stack, a central credential management solution, centralized policy enforcement of specific Kubernetes resources, and the like.

The lower end of the spectrum will likely not be managed by GitOps. That is simply because the GitOps tooling itself typically needs to run somewhere, and also needs to be bootstrapped somehow. Some tools such as FluxCD allow the GitOps controller to manage itself, but even in these cases the runtime for the controller needs to exist when the controller is initially installed, and is thus typically not part of the GitOps configuration.

Now that this is cleared up, let us consider how the configuration should be managed.

App-of-Apps

A very popular pattern for managing configuration via GitOps is the "app-of-apps" pattern. This was popularized by ArgoCD, but is also applicable to other tooling. We will use ArgoCD in the example below, but the same can be implemented using FluxCD Kustomizations.

Let us consider a component from our infrastructure that we want to manage via GitOps. Typically, we would need to tell the GitOps controller how to manage this component. For instance, let us assume the component is installed via raw Kubernetes manifests. Then we would tell the GitOps controller which repository contains these manifests and in which namespace to install them. Depending on the controller you are using, you might also configure additional parameters such as how often it should be reconciled, whether it depends on other components, and so on. In ArgoCD jargon this would be an "Application" (the root of "app-of-apps" naming), and would look as follows:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
spec:
  project: default
  source:
    chart: sealed-secrets
    repoURL: https://bitnami-labs.github.io/sealed-secrets
    targetRevision: 1.16.1
    helm:
      releaseName: sealed-secrets
  destination:
    server: "https://kubernetes.default.svc"
    namespace: kubeseal

You would then apply this Application resource to Kubernetes. Your component would then be managed by GitOps, as any changes you push to the manifests repository would be reflected on the Kubernetes cluster.

Then a second infrastructure component needs to be installed, and you repeat the process. The result would be a second Application which installs and manages a component. You might also want to version your deployment (such as using version 1.16.1 of the Helm chart). This implies that lifecycles require a change to this Application manifest, and thus a call against the Kubernetes API to edit it.

The end result is a set of Application resources, some of which you periodically modify when lifecycling a component. Now imagine you need to deploy your infrastructure elsewhere (for instance a second Kubernetes cluster in our example), or maybe even a couple dozen times. Then you need to manage this entire set of Application resources on every platform. A better approach is to add an abstraction layer, which itself deploys the Application resources via GitOps. Hence you put all your Application resources into a repository, and define another, "higher level" Application which deploys this repository. This means that when deploying to new platforms, you only need to deploy that one "higher level" Application, and any changes to the component Application resources can be made via Git, conforming to our GitOps approach. This "higher level" Application is only there to deploy the component Applications thus the name "app-of-apps". Visually, you thus have the following structure:

It should be noted that this also massively helps when customizing platforms. Typically, components cannot be deployed truly one-to-one in several places, but require slight configuration differences. Consider for instance hostnames for UIs of your components. Two of these components deployed in different locations cannot share the same hostname and routing. Using an "app-of-apps" approach allows you to define variables on the top level application, and inject these into the downstream applications such that they can slightly adapt the way they are installed. We will not dive deeper into how this is done as it is highly dependent on the tooling you use (ArgoCD uses ApplicationSet, FluxCD uses variable substitution), but know this is enabled by such an approach.

Consolidating your Configuration

In the organisation I first used GitOps at scale, we deployed all our components as Helm charts to a Kubernetes cluster. Each component was essentially contained within two different repositories in our version control system:

the source code repository which typically built a Docker image as an artefact
the Helm chart definition which referenced the Docker image from above

When we then introduced GitOps, we decided to add a third repository containing the exact deployment definition (in our case the Application declarations) for the component. Using the app-of-apps pattern from above, we could then reference each of these "GitOps repositories" and deploy specific overlays (customizations) of the Application to specific platforms. This worked well for quite some time. However, with time the number of components we managed increased, and so did the number of target platforms to which these components needed to be deployed. This lead to quite a few issues.

When a new target platform was introduced, all such "GitOps repositories" needed to be updated to contain a new overlay customizing the Application to the specific platform. This is very tedious when you have several dozen such repositories.

Moreover, components had dependencies to other components. This meant that we were referencing components within a repository that were defined in another repository. While not problematic in itself, this can become very tricky when one component has a dependency on a configuration value of another component. The configuration value is then duplicated in both repositories and becomes difficult to maintain. While this sounds like we did not properly separate the components, it is very common to see such cases in infrastructure configurations. Consider for instance a deployment of an ingress controller which defines a hostname suffix for its routes. All components deployed on the same Kubernetes platform that deploy a route/ingress will need to use exactly that hostname suffix in order to have valid routing.

The above issue also results in tricky situations when configurations need to be changed for components that are dependent on one another. If the deployment configuration is separated into different repositories, PRs to these repositories need to be synchronized to ensure the deployment occurs at the same time.

Finally, distributing the deployment configuration over so many repositories meant that it became increasingly difficult to have an overview of what is deployed on a target platform. One would need to navigate through dozens of repositories to check this is correctly done.

After identifying these issues we decided to move all our configuration into a single repository. This repository would then contain a templated definition of the entire set of components which would need to be deployed. A set of platform definitions within the same repository would then feed values to templates to ensure consistent configuration. This massively helped us with to address the issues mentioned above. On top of that, it allows to version the "template" and thus enables rollouts of a versioned infrastructure layer.

You can find an example repository of such a structure
designed with FluxCD here:

f4z3r / flux-demo

This repository shows an example how one can use a single mono-repository to manage multiple clusters' infrastructure in a controlled fashion.

Flux Demo

An example how one can use a mono-repo to manage large infrastructure in a controlled fashion using FluxCD.

Setup | Structure of the Repo | Application vs Infrastructure | Workflow

Setup

Generate a GitHub PAT

See the documentation: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens

For fined-grained control, grant the token Admin and content read/write permissions on the repository.

Setup a Cluster with Flux

Setup the required tooling with devbox shell, then

# install a cluster
kind create cluster -n demo
# set the token
export GITHUB_TOKEN='<redacted>'
# onboard flux
flux bootstrap github \
  --token-auth \
  --owner=f4z3r \
  --repository=flux-demo \
  --branch=main \
  --path=clusters/demo \
  --personal

Sample output from Flux installation

► connecting to github.com
► cloning branch "main" from Git repository "https://github.com/f4z3r/flux-demo.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed component manifests to "main" ("158753158f3c760f741f22ed7f68bdee1b66e475")
► pushing component manifests to "https://github.com/f4z3r/flux-demo.git"
► installing components in

…

View on GitHub

Gitops Bridge

The last challenge we want to address in this blog post is a concept called a "GitOps bridge". In public cloud environments, there is typically a relatively strong cut between infrastructure deployed via Terraform (or any similar tool), and the infrastructure deployed via GitOps. For instance, one might deploy an Azure Kubernetes Service and some surrounding services (such as the required network, a container registry, etc) via Terraform, and them deploy components and applications within the AKS using GitOps. The issue that we face here is that the GitOps configuration very often depends on the Terraform configuration. Consider for instance the container registry. Its address is set up by Terraform, but is used in every image declaration in the GitOps configuration. One option is to duplicate such values in the respective configurations, while another option is to use a GitOps bridge.

The GitOps bridge is an abstract concept on how to pass configuration values from tooling such as Terraform as inputs to the GitOps configuration. How this is done in practice very much depends on which tools you use. For instance, if looking at Terraform and FluxCD, a common way to achieve this is to have Terraform write a ConfigMap onto the AKS where the FluxCD controller will run containing all variables (and their values) that will be required by the GitOps configuration. The FluxCD controller then supports injecting variables from a ConfigMap via variable substitution.

Using a GitOps bridge has the advantage that changes in the Terraform configurations are much less likely to break the GitOps configuration that builds on top of it. Moreover, it allows Terraform to directly bootstrap the entire GitOps setup when creating new platforms without the need to manually redefine the required variables in the GitOps repository.

Summary

So, to recap, we have looked at what GitOps really is (and isn't). Understanding these basics is critical to correctly implement GitOps in your projects. On top of that, we looked at three best practices:

Use an app-of-apps pattern to improve resiliency for when you need to recreate platforms.
Consider using a mono-repository for all your GitOps configuration as your setup grows.
Have a look at GitOps bridges to improve the automation when setting up platforms and ensuring your Terraform and GitOps configurations are consistent.

I hope this has helped you understand a bit better how to use GitOps at scale. If you have any questions or comments, feel free to let me know below.

A Very Deep Dive Into Docker Builds

Jakob Beckmann — Tue, 26 Nov 2024 06:25:41 +0000

Containers are everywhere. From Kubernetes for orchestrating deployments and simplifing operations to Dev Containers for flexible yet reproducible development environments. Yet, while they are ubiquitous, images are often built sub-optimally. In this post we will be looking at a full example of a Docker build for a Python application and what best practices to consider.

Disclaimer

This is a real world example from an very small component we built in Python for one of our clients. Very few alterations were made to the original configuration (changing URLs, and removing Email addresses mostly). We will go in depth as to why we did every single little thing. While some stuff is quite Python-centric, the same principles apply to other languages, and the text should be broad enough so that it is understandable how to transfer this example to different languages.

Also, this is a long article, so if you actually plan on reading it, grab yourself a snack and a fresh drink first.

Goal

The goal of this post is to showcase how one can setup a Docker build that is:

fully reproducible,
as fast as possible,
fails early on code issues,
isolates testing from deployed code,
is secure.

The example we will use for this implements quite a lot to ensure only quality code reaches production, and that it can do so as fast as possible. Going all the way might not be necessary for all projects using Docker. For instance, if you release code to production only once a day (or less) you might care less about release build cache optimization. This example is however meant to show the "extreme" to which you can push Docker, so that you can (in theory) push code to production fully continuously (CI/CD principles). But yeah ...

Why

Why do we have these goals? Reproducible builds are one of the most important factors for proper compliance, and for easier debugging. Debugging is simpler, since we ensure that no matter the environment, date, or location of the build, if it succeeds, the same input generates the same output. Moreover, it brings stability, as a pipeline might not suddenly fail on a nightly build (if you still do such things) because a new upstream library or program was released that is used somewhere in your supply chain.

Regarding compliance, we need to be able to tell and revert to the exact state of software that was deployed in the past. Without reproducible builds, using Git to track back to a previous state of deployed code does not help you much, because while you can validate what code you deployed, you don't know what versions of everything else you deployed with it.

Builds should be fast, and fail fast. The reason here is that no one likes to wait. You don't want to wait for 2 hours to figure out whether a tiny code change breaks tests or does not even compile.

You will want to isolate test code from deployed code, because more code equals more bugs. While testing frameworks are very good at isolating test code from code being tested, writing tests generates a risk of bugs. Moreover, the test code is unneeded bloat for your runtime application. Thus it should be isolated from it.

Finally security. While some people think that containers improve security by default, this is not the case. Container technology has the potential to indeed improve the robustness of some security measures and controls. However, in order to achieve this, one needs to correctly utilize containers and build the images with security in mind. For instance, if an image contains certain utilities that allow it to connect to the internet (such as curl or wget), it suddenly makes the container much more vulnerable to container escape attacks (where an attacker manages to move from the container to the underlying host), and hence the whole isolation benefit of the container (which can be a security control) is broken. The same is true for containers that contain interpreters and allow the runtime user to open, edit and execute arbitrary files. As our container will contain Python code, and hence the Python interpreter, this is definitely something we need to take very seriously.

Python Goals

Our example is based on Python, an interpreted language. This is not ideal, as it means that it does not require a compile step. Compilation optimization is however a very important aspect in Docker builds. In order to still address this, I will talk about this, but will not refer to the configuration examples. One could ask why I did not take a compiled language example then. The reason is very simple, I wanted a real world example such that this post is not just theoretical goodness, and most Golang image builds I am currently working on are more basic and not as educational.

Yet another question could be "why deploy Python in Docker in the first place?". This is a very legitimate question. Python requires a lot of OS bloat to just be able to run. This means that typically a VM is a good choice to host it. For all those saying that Docker is still better because of performance (due to faster startup, no hardware virtualization overhead, etc): this is not true for such cases where a large part of an OS needs to be in the Docker image. A VM of a full init-based Linux system can be launched in less than 250ms on modern technology. A full Ubuntu installation with systemd can be completely booted in around 2.5 seconds. The former is in the same order of magnitude that it might take the Python interpreter to just load the code of a large Python application.

So performance cannot be said to be better with Docker, why choose Docker then? Better reasons are that you can strip down a Docker image much easier than an OS. This is critical for us due to security requirements. While Python requires a lot of OS features, the majority of the OS is still bloat. Every piece of bloat is a potential attack vector (each of these unused components might have one or more CVEs that we need to patch, even though we don't even use that software). Another reason is that the build process of Docker is much simpler to manage. There are tools such as Packer that allow similar processes for VMs, but these are not as standardized as the open container initiative (OCI - which Docker adheres to).

Another very important point is the ease of development. Docker and other OCI compliant products provide us with a possibility to build, test, and run our build artefacts (in this case Docker images) everywhere. This makes it very simple and fast for our developers to test the build and perform a test run of an image locally on their development machine. This would not be quite the case with VMs or raw artefacts (JARs, source code archives, ...). Moreover, the OCI ecosystem does not only include specifications on how to interact with images, but also how to setup and configure critical elements such as persistence and networking. These aspects are made very simple with Docker, and would be quite a pain to manage securely with most other technologies.

Finally the main reason for us is the choice of runtime. We have very decent container runtimes (RKE, RHOS, K3s) available to deploy applications. We are very familiar with them, and they offer us a lot of functionality. These all support containers primarily.

A Tiny Bit of Background

Last before we get into the dirty details, a tiny bit of background into what we are building. The application we will be building here is a sort of a facade reverse proxy. It offers a standardized API to clients, which can connect and perform requests. Based on the content of the request, the component will trigger a routing algorithm that defines where the request needs to be routed. This routing algorithm might require several API calls in the backend to different systems to figure out where the call should go. Once done, the component will relay the call to a backend, and forward the response to the client. The client is never aware that it is talking to more than one component, and only needs to authenticate to that single system. Imagine an API Gateway, but where the routing is extremely complex and requires integration with systems such as Kubernetes, a cloud portal, and more.

The Details

Here is an overview of our Dockerfile:

FROM internal.registry/base/ca-bundle:20220405 AS cert-bundle

FROM internal.registry/base/python:3.9.2-slim AS builder

COPY --from=cert-bundle /certs/ /usr/local/share/ca-certificates/
RUN update-ca-certificates

WORKDIR /app

RUN pip install \
  --upgrade \
  --no-cache-dir \
  --ignore-installed \
  --trusted-host pypi.python.org \
  --trusted-host pypi.org \
  --trusted-host files.pythonhosted.org \
  pipenv==2024.2.0

ENV PIPENV_VENV_IN_PROJECT=1
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

COPY Pipfile Pipfile
COPY Pipfile.lock Pipfile.lock

RUN pipenv install --deploy

### Tester image
FROM builder AS test

RUN pipenv install --dev --deploy

COPY ./pyproject.toml pyproject.toml
COPY ./assets/ ./assets
COPY ./features/ ./features
COPY ./tests/ ./tests
COPY ./src/ ./

RUN --mount=type=cache,target=./.mypy_cache/ \
  --mount=type=cache,target=./.pytest_cache/ \
  pipenv run mypy . \
  && pipenv run black --check . \
  && pipenv run bandit -ll ./*.py \
  && PYTHONPATH=./ pipenv run pytest


### Runner image
FROM internal.registry/base/distroless-python:3.9.2
LABEL maintainer="Redacted <redacted-email>"

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

WORKDIR /app/
USER 1000

COPY --from=builder --chown=1000 /app/.venv/lib/python3.9/site-packages ./my-app
WORKDIR /app/my-app

COPY --chown=1000 ./src/ ./

ENTRYPOINT ["python3"]
CMD ["./main.py"]

We will go through it line by line and figure out why we did what we did, and why we did not choose a different approach. Let's start!

FROM internal.registry/base/ca-bundle:20220405 AS cert-bundle

In this line we reference a container image for later use and provide it a name alias cert-bundle. This container image contains only data: our production network proxy certificates and all internal certificate authorities. We need these CAs as we will connect over TLS to backend components that have internal certificates. We also need the production network proxy certificates as we will pull dependencies straight from the internet, and all that traffic is routed over a gateway proxy. Why distribute these certificates over a Docker image instead of a compressed TAR? The main reason is that we want to have a unified way that we build artefacts and manage CI/CD pipelines. By creating and managing the certificates via Docker, we can use our entire Docker setup (such as UCD/Jenkins/Tekton pipelines for building, registry for distribution, quality gates for security, etc) and do not need to have a different system to manage the certificates. Note that we refer to the exact state of the certificate bundle (20220405), which refers to the state of the certificates per 5th of April 2022. This is very important to make the build reproducible. If we did not pin the version of the certificates, it would mean that we could build the image maybe today, but it would fail tomorrow, once the certificates change (even though we did not change the code at all). You will note that we will pin every single version in the entire build process.

FROM internal.registry/base/python:3.9.2-slim AS builder

In this line, we reference the base image we will start building from. This is the official Python image for Python version 3.9.2. We use the slim version because we don't need much more than the standard Python installation. We pull this from our own registry, as all Docker images are scanned beforehand to reduce the risk of supply chain attacks. Also here, the version is pinned. We provide this build step the builder alias. In essence this means that starting from this line we define an image stage that will contain the build process of our application. For Python, this mostly includes downloading dependencies (both software and system level), and injecting the source code, as there will be no compile step.

COPY --from=cert-bundle /certs/ /usr/local/share/ca-certificates/

This copies our certificates into our build image. We do this by referencing the build step cert-bundle (see first line of the Dockerfile again) in the --from argument of the COPY command. Note that we could have referenced the image directly in the --from argument. We choose to use build stage aliases for visibility, and reduce duplication if the certificates need to be copied into different stages. Note that this copies only the raw certificates. A OS specific bundle would still need to be generated.

RUN update-ca-certificates

Here we do exactly this, we generate a certificate bundle for the underlying OS of our builder image (Debian). This allows our subsequent build steps to use the certificate bundle to validate host certificates on TLS connections.

WORKDIR /app

We then set a working directory. The idea is to have a base directory on which we now operate. This can be nearly any working directory, and will be created if non-existent. We choose /app/ by convention. Moreover, note that we tend to reference directories with the trailing / to make it more explicit that we are referencing directories and not files. We use this convention throughout the configuration/code.

RUN pip install \
  --upgrade \
  --no-cache-dir \
  --ignore-installed \
  --trusted-host pypi.python.org \
  --trusted-host pypi.org \
  --trusted-host files.pythonhosted.org \
  pipenv==2024.2.0

We use an environment virtualization technology for Python. This is called pipenv. It allows us to have many different versions of the same dependency installed locally, without them conflicting. This is very important when you are developing many applications at the same time locally. By running this line we install version 2024.2.0 of pipenv (pinned). Other than Python itself, these are the only tools required for our Python development environment. If we were using a different language, pipenv would be substituted with your dependency management tool (such as Maven for Java). Note that we only install pipenv itself, we do not install the dependencies. Also using the flags provided we ensure a fully clean install of pipenv.

This is an example where we reach out to the internet and thus needed the network proxy certificates.

A very good question here might be "why to use pipenv at all, considering it is typically used for environment virtualization, which is already covered by Docker itself?". There are two aspects here. The first is to allow us to lock dependencies using their hash, which is not natively supported by pip (the standard Python package manager). The second is that we want to keep the build process within Docker as close to the build process outside of it. While we do not build artefacts outside of Docker per-se, the IDEs of our developers need to fall back on these technologies to support features such as library-aware code completion, type-checking, test integration, debugging, etc. This could also be achieved by connecting the IDE to an instance running directly in Docker. This however is relatively complex and requires the setup to support remote debugging. In theory, these are not really problems as long as the dev environments are uniform, but we allow each developer to work with the tools he/she desires to develop code. It then suddenly becomes very difficult to have a stable setup that works for everybody, especially considering that some of our developers do not want/know how to configure their environments to that level (client-server debugger setups, network and volume management between the IDE and Docker, ...).

ENV PIPENV_VENV_IN_PROJECT=1
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

Here we set environment variables for pipenv. Firstly we want the dependencies to be installed directly in the project repository, not centrally. This allows us to ensure that we do not accidentally copy a system Python dependency that installed by default with the base image. The second configures the certificate bundle we generated in the beginning to be used by pipenv. It does not use the system configured bundle by default, so it needs to be configured manually here.

COPY Pipfile Pipfile
COPY Pipfile.lock Pipfile.lock

Now the interesting stuff. Here we copy the dependency files into the image. The first file contains a list of dependencies that we use for our project. The second contains a hash the dependencies should have, including indirect dependencies (dependencies of dependencies), in order to ensure that we always get exactly the same dependency code for very install. The first looks as follows:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "==2.28.2"
pydantic = "==1.10.4"
# more dependencies ...

[dev-packages]
black = "==23.1"
bandit = "==1.7.4"
pytest = "==7.2.1"
pytest-mock = "==3.9.0"
pytest-bdd = "==6.1.1"
mypy = "==1.1.1"
types-Pygments = "==2.14.0.6"
# more dev dependencies ...

[requires]
python_version = "3.9"

Note that we split dependencies into normal packages we require for our application, and packages only required for testing and our quality gates ([dev-packages]). This is important later on, as we do not wish to have packages only required for testing in our production Docker image.

I will not show you an example of the lock file, as it contains mostly checksum hashes. Simply trust me that it contains the exact checksum that every package (such as the dependencies of requests) has to have to be installed. The reason this is required in the first place, is because the dependencies of requests are likely not pinned to an exact version and might thus change between installations unless locked via our Pipfile.lock. This would undesired as it would make our builds un-reproducible. The lock file itself is generated by our developers in two different scenarios. The first is when a library is added due to some new feature. In such a case the new library is added to the Pipfile, and an installation is triggered outside of Docker. This will install the new library and potentially update already installed ones (in case of conflicts). Hence new hashes will be added to the lock file. The second is on a lifecycle of the existing libraries or of our Python version. In such a case we update the pinned version in the Pipfile and trigger an installation outside of Docker. Again, pipenv would then update the direct dependencies, and potentially transitive ones, and update their hashes in the lock file.

RUN pipenv install --deploy

Here we install the dependencies for our application. The --deploy flag means that we want to install the dependencies based on the lock file. Moreover, we do not install the dev packages yet, only the ones needed for the production code.

### Tester image
FROM builder AS test

Here we generate a new Docker build stage. We have generated a stage with builder that contains the required certificates and the production dependencies, and nothing more. We now want to test our code and validate quality gates. We do not want to perform this in the builder stage, because it would pollute our production dependencies. Moreover, using a different stage allows to trigger builds more granularly with BuildKit. For instance, I would be able to configure (with --target=test) to only build the image up to the test stage, and skip any later stages (such as the runtime image in our case). This can be very useful in pipelines, for instance, where we want to run the test on every commit, but are not interested in building a real artefact unless the commit is tagged.

With this line we essentially say "start a new stage called test from the latest state of builder". We also add a comment above to make it more visible that we are starting a new stage in the Dockerfile. Stage comments are typically the only comments we have in the Dockerfiles.

RUN pipenv install --dev --deploy

In this line we now deploy the development dependencies, including tools for quality checks (mypy, bandit, black, see below for details) and for testing. Again, we use the --deploy flag to ensure we always use the same versions to make the build fully reproducible.

COPY ./pyproject.toml pyproject.toml
COPY ./assets/ ./assets
COPY ./features/ ./features
COPY ./tests/ ./tests
COPY ./src/ ./

Here is the first time we copy actual content, other than the list of dependencies, into our image. This means that up until now all layers in the build process can be fully cached if we perform code changes. Thinking about this is primordial if you want an efficient build process. Even here, we copy code in reverse order based on likelihood of change. The first file we copy in configures our tooling and quality gates. This is unlikely to change unless we introduce a new tool or change configuration of an existing one. An example of the file can be seen below.

The second line copies assets. These are used for testing, such as test configurations for configuration validation etc. These are also quite unlikely to change unless we write new tests of our configuration classes.

The third line copies in our Cucumber files for BDD testing. These change only when we either define new behavioral tests or add features.

The fourth line copies our test code, this is quite likely to change, as it contains all our unit tests, and the testing framework for behavioral tests.

Finally the last line copies in our actual code. This, along with the unit tests, is the code that is most likely to change, and thus comes last. This way on a code change, all lines up to this one (assuming we did not add/change tests) can be used from cache.

[tool.black]
line-length = 100

[tool.pytest.ini_options]
pythonpath = [
  "src",
  "tests",
]
bdd_features_base_dir = "features/"

[tool.mypy]
exclude = [
  '^tests/.*\.py$',
]
ignore_missing_imports = false
warn_unused_configs = true
warn_redundant_casts = true
# more settings ...

[[tool.mypy.overrides]]
module = [
  "kubernetes",
  "parse_types",
]
# skip libraries without stubs
ignore_missing_imports = true

RUN --mount=type=cache,target=./.mypy_cache/ \
  --mount=type=cache,target=./.pytest_cache/ \
  pipenv run mypy . \
  && pipenv run black --check . \
  && pipenv run bandit -ll ./*.py \
  && PYTHONPATH=./ pipenv run pytest

This line aggregates our quality gates and testing. For quality gates we have:

mypy: checks typing information where provided. We do not perform strict typing so that type information is required everywhere, but we validate that the provided typing is correct.
black: checks formatting of the code to ensure it is according to your guidelines.
bandit: performs basic security checks. This is a non-blocking check, meaning that the build will only fail if issues of severity MEDIUM or higher a found. LOW severity check fails are ignored.

Finally we run our testing (with pytest). We run the testing last, as it is the most time consuming of the tasks, and it does not need to be executed if the code fails to adhere to our standards. Note that you could add any other gates here, such as a code coverage baseline that needs to be adhered to, various code analysis checks, or security scans. We only perform one more security check against dubious code and supply chain attacks during the build process. This check is however done on the final Docker image and is thus executed by the pipeline itself outside of the Docker build process.

Note that all commands are executed as one RUN statement. This is best practice, as none of these commands can be cached individually. Either all have to be executed again if layer it builds upon changed, or none has to run. Putting them into the same RUN statement generates a single new layer for all four commands, which reduces the layer count and build overhead for Docker.

Finally, note the --mount options passed to RUN (introduced with BuildKit 1.2). These allow to cache content within the Docker build between builds. Here we mount two caches, one for mypy and one for pytest. These ensure that if a subsequent Docker build is triggered for code that does not affect some files, the typing checks and tests are not run again for these files, but taken from the cache. For pytest this is actually done on a "per-test" basis, ensuring tests are not run unless code they are testing is changed. Such caches can massively increase the speed of your pipelines, especially when your project grows and the test suites start to take more time to run through.

### Runner image
FROM internal.registry/base/distroless-python:3.9.2

This defines the runner image. We are done with testing and want to build the productive artefact, as all checks have passed. In a compiled setup, this would mean we would now have a release compilation stage (before building the runtime image). This is done after testing as the release binary/JAR will be compiled with optimizations, which can take quite long, and is unnecessary if the tests fail anyways. Thus in a compiled language like Java or Golang, we would now continue from the builder again, copy the code back into the layer, and compile. Here one should be careful, most languages support incremental compilation to reduce compilation times. When this is supported, one needs to mount a build cache, or the incremental compilations from previous builds will be lost every time the code changes, as the entire compilation layer will be discarded from the cache. This is done the same way as in the previous block, with --mount parameters.

Once the compilation is completed, and we have our final artefact (binary or JAR), we want to copy it into the runtime image. The idea is again to restrict bloat to reduce our attack surface. For instance, in a Java setup, we only need a working JRE to run our application, we no longer need Maven, the Java compiler, etc. Thus, after the build process, we use a new stage for the runtime image. This is what we did for Python here, since we have no compilation step. We use a different image than our initial internal.registry/base/python:3.9.2-slim image, as we no longer need pip (the Python package manager), and other bloat. Instead we use a distroless image, which is essentially a stripped down Debian image containing truly the base minimum to run Python code, but nothing to manage it, etc. Again, we use our own copy of the distroless image from our scanned registry.

LABEL maintainer="Redacted <redacted-email>"

This line adds metadata to the image. This is not necessary to have a good image, but useful when using images that are shared across huge organisations. This is the official maintainer label we use, where we reference our team, such that anyone that downloads the image and inspects it can see who built it, and how to get into contact with us in case of issues.

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

Same as before, we copy certificates and configure Python to use our bundle. Note that this time we directly copy the bundle generated in the builder and not from the certificate image, as we need a bundle and cannot create it in this image (update-ca-certificates is not contained in the distroless image). We need to copy this explicitly since we started from a fresh image. The test stage had the bundle implicitly configured from the builder stage, upon which it was set up.

WORKDIR /app/
USER 1000

We set a working directory again. This is also necessary since starting from a fresh image. Also we set a non root user. This is necessary since we do not want to run our code as root for security reasons (reduce the impact of a remote code execution - RCE vulnerability). Note that any statement after the USER statement will be executed in the context of that user. Therefore I would for instance not be allowed to run update-ca-certificates (if it was present in the image) in a RUN statement from now on, as this requires root privileges.

COPY --from=builder --chown=1000 /app/.venv/lib/python3.9/site-packages ./my-app
WORKDIR /app/my-app

Here we copy the non-dev packages from the builder stage into our productive image. Note that we use a path from within the project root (/app/), since we set pipenv to install the virtual environment directly in the project (the PIPENV_VENV_IN_PROJECT variable). We copy the site-packages (the dependencies) directly into a subfolder, in which our application will live. This ensures that they are treated as if we wrote them ourselves, as individual Python modules in our code. They essentially become indistinguishable from our own code. This allows to keep consistency in our module names are resolved. Note we need to add the --chown flag, as the dependencies were installed by the root user in the builder image, and they need to be readable by our user 1000 that will run the application. The --chown flag will change the files' owner (and group) to the provided argument.

The second line simply sets the new working directory to be the new project directory into which we copied the code from the dependencies.

COPY --chown=1000 ./src/ ./

Here we copy the source code back into the production image. We did this after copying the dependencies, such that the dependency layer can be cached again. Moreover, we only copy the source code, no tests, no assets, no Cucumber features. All these latter ones are not needed to run our application. Finally note that we copy it not from the test stage, but again back from the outside build context. This is because we mock a lot during testing, changing some code behavior dynamically. Copying it back in from the outside context ensures we do copy the exact code that is in our Git repository, and not something that was accidentally modified during testing, etc.

ENTRYPOINT ["python3"]
CMD ["./main.py"]

Finally we set an entrypoint and a command. The entrypoint defines what will always be executed on a Docker run (unless explicitly overwritten), and the command provides the default arguments unless overwritten via the Docker run arguments. We always use lists instead of full strings to ensure that the arguments get passed to the Kernel as system calls instead of being executed by a shell. This is important to ensure proper signal handling (when you want to terminate containers), and since there is simply no shell in the distroless image we are building.

That's it

Holy molly... There is a lot that goes into building a simple Docker image. And that considering we did not even compile anything, which would require a decent amount of extra work, and that all our tooling can be managed directly via pipenv and do not need to be installed separately via curl or some OS package manager.

So is it worth it? To put so much thought into how a simple Docker image gets built? I would argue yes. I will not start an idiomatic discussion on the benefits of smaller images, security best practices, or having tests being run directly in the Docker build. If you want such a discussion, go to Reddit or Youtube, you find plenty of beef between people fighting about these topics like their life depends on it. All I will say is this:

I can run docker build ... after each save on a file, since the caching is optimized to a point where a full build on a code changes takes about 1-2 seconds. Being able to run this so often gives me the confidence that what I will push will actually pass in the pipeline.
Using proper caching makes me avoid having to wait 2-5 minutes each time I want to compile something. Since 2-5 minutes is typically too little for a context switch to something else, it might be time I would have just sat around thinking about how much it sucks to wait on stuff. So it has considerably improved not only my productivity, but also my mood.
Docker avoids some "it works on my machine" issues. With proper version pinning and fully reproducible builds, it really nearly eradicates the issue. Now the only time something like this can happen is when running on different Docker versions.
We all sometimes would like to fix tests by skipping them to "save time" when something needs to go to production quickly. Since testing is fully baked into the build process, changing flags on Jenkins/Tekton/whatever will not allow you to skip any testing or quality checks on the code. The only way would be to comment out the test code or update the Dockerfile, which would not pass a PR review. This gives me immense peace of mind.

Since the build process and testing is (nearly) fully defined in the Dockerfile which lies in the git repository, we nearly never need to change pipelines to add/change/remove anything, as all of this can be done in the repository of the corresponding image directly. This also has downsides, as it creates duplication. I would argue that this is beneficial though, as legacy applications might not be able to switch to newer tooling as fast as greenfield projects, which want to leverage that new tooling. Having this "configured" in each repository allows each to move at its own pace. Strict guidelines (such as we don't want to use tool X anymore) can still be enforced on pipeline level via container scanning tools (which you will need either way).

What's the major downside of this approach? Well I would argue there is one large one. Many people might not understand Docker well enough to figure out how the build process works, or might not have time to invest to learn how to do it correctly. This means that some people might not be able to make changes to the build processes by themselves and need might help. I think this would also be the case without a proper Docker setup, but maybe this problem is augmented by having a slightly more complex Docker build setup.

I hope this has given you some food for thought. Feel free to comment any questions or remarks below, or to reach out! Do you also take your Docker builds this far?

Jakob BeckmannFollow

I am passionate about cloud native software and infrastructure, security tooling, and various programming languages. I am currently a Principal Architect at ipt.ch, working on a variety of mandates.

Innovation Process Technology AG (ipt) Follow

We are a boutique IT consultancy based in Switzerland focused on building individual solutions using leading edge technology. For more information visit our website: https://ipt.ch

DEV Community: Jakob Beckmann

Beyond the Pod: Why wasmCloud and WebAssembly Might Be the Next Evolution of the Platform

WebAssembly as a Platform Foundation

WebAssembly: The Component Model

WebAssembly: Secure by Default

WebAssembly: Performance

wasmCloud Architecture

Hosts

Lattice

Capabilities

Providers

Components

Applications

wadm

Verdict

Dissecting Kubewarden: Internals, How It's Built, and Its Place Among Policy Engines

Policy Engines

Kubewarden Architecture

Comparison

Verdict

The Tortoise and the Hare: do AI Agents Really Help for Software Development?

I am an AI Skeptic

AI Support Agents

CodeRabbitAI

Starting Small

A More Complex PR

Dashboards!

Verdict

f4z3r / gruvbox-material.nvim

Material Gruvbox colorscheme for Neovim written in Lua

Gruvbox Material

A NeoVim colour scheme in pure Lua allowing for highly flexible configuration and customization.

Features

f4z3r / sofa

A command execution engine powered by rofi.

Sofa

A command execution engine powered by rofi and fzf.

About

Examples

For Snippets Management

A Comprehensive Guide to Managing Large Scale Infrastructure with GitOps

GitOps Overview

GitOps is More than CI/CD Pipelines

The Advantages of GitOps

Tooling

Infrastructure: Disambiguation

App-of-Apps

Consolidating your Configuration

f4z3r / flux-demo

This repository shows an example how one can use a single mono-repository to manage multiple clusters' infrastructure in a controlled fashion.

Flux Demo

An example how one can use a mono-repo to manage large infrastructure in a controlled fashion using FluxCD.

Setup

Generate a GitHub PAT

Setup a Cluster with Flux

Gitops Bridge

Summary

A Very Deep Dive Into Docker Builds

Disclaimer

Goal

Why

Python Goals

A Tiny Bit of Background

The Details

That's it

Jakob BeckmannFollow

Innovation Process Technology AG (ipt) Follow

A command execution engine powered by `rofi` and `fzf`.