DEV Community: Camptocamp Infrastructure Solutions

Platform Engineering at CNS Munich

Xavier Rakotomamonjy — Tue, 26 Aug 2025 14:33:58 +0000

This year Cloud Native Summit (CNS) in Munich gathered adopters and technologists from open source and cloud native communities.

Compared to last year's Kubernetes Communities Days (KCD), the agenda dedicated many slots to Platform Engineering. This discipline can be seen as a way to fill gaps between different stakeholders of an Internal Developer Platform (IDP). Last year I wrote an article on this topic at the KCD Munich. This year I would like to share fresh ideas or principles from some talks given during the conference.

Identifying problems before solutions

Day 1 opened with a presentation of "Product thinking" approach made by Stéphane Di Cesare (Deutsche Kreditbank) and Dominik Schmidle (Giant Swarm). They reminded some core principles and techniques of "Platform Engineering" to manage the link between problem space and solution space.

Day 1 also closed with a panel discussion on "Platform Engineering" animated by Max Körbächer (Liquid Reply), focused on the adoption of IDPs by end users and within projects.

Video 1 Platform Thinking

Video 2 Panel Discussion

AI workloads on K8s

"Use platform engineering for AI workloads on K8s", that was the motto of the presentation given by Mario-Leander Reimer (QAware GmbH). He tackled emerging challenges for deploying Agentic AI workloads and introduced an example of what a platform should cover including quality plane and compliance plane. His talk is part of an open source initiative launched at QAware this year to mature the concepts of a dedicated control plane: the agentic layer.

Video 3 Architecting and Building a K8s-based AI Platform

Composable platforms on Kubernetes

Hossein Salahi (Liquid Reply) presented a reference implementation to streamline the delivery of services for developers. Some of the challenges were to link dev and ops, to manage services in configuration, to orchestrate infrastructure deployment and to manage access. For that purpose the open source tools Kratix codify a contract between dev and ops. Combined with Backstage, this abstraction allows services to be packaged and delivered to developers.

Video 4 Modular Platform Engineering with Kratix and Backstage

Reduce cognitive workloads

Interestingly the problem of cognitive workloads due to some platform complexity was a recurring theme. But Michel Murabito (Mia Platform) positioned platform engineering as a way to reduce cognitive workloads. He highlighted essential building blocks and features that can reduce mental burden.

Video 5 Creating a smooth Developer Experience

Involved user as contributor

Lian Li (lianmakesthings) told us a story of what can go wrong when building a platform based on real examples and some facts. She advocated some receipts that can improve communication among teams in an organisation and explain some benefits of having contributing users in the loop.

Video 6 Many Cooks, One Platform: Balancing Ownership and Contribution for the Perfect Broth

Convergence

Evelyn Osman (enmacc) discussed the importance of convergence when building a platform in order to avoid fragmentation in teams and solutions. She shared some important steps that foster innovations and better meet user expectations.

Video 7 Convergence on Platforms

Conclusion

This is not an exhaustive list of what was discussed, but somehow it made it clear that Platform Engineering approach can help picking the right solution in the open source landscape. Some enablers already provide a strong integration with Kubernetes to develop and publish new services on top of it. And depending on the organization, some shift in practices also might be required to follow the evolution of the platform at build and runtime. The conference was a great place to exchange with the community on technical and organization challenges.

References

Talos Linux: a new standard for on-premises Kubernetes clusters?

Gonçalo Heleno — Tue, 22 Apr 2025 17:44:21 +0000

A few weeks ago, I was at KubeCon Europe 2025 in London and I had the opportunity to attend a presentation that tackled the monumental challenge of migrating 35 Kubernetes clusters in an air-gapped environment from nodes deployed with a mix of kubeadm/Ansible/Puppet to Talos Linux nodes deployed using Cluster API.

While the presentation was quite interesting (you can get the slides here and watch the session recording on CNCF's YouTube Channel), I want to dive more into the Talos Linux project and its features.

What is Talos Linux?

Talos Linux is a modern Linux distribution purpose-built for running Kubernetes clusters. Some noteworthy characteristics of Talos Linux are:

Immutable: Talos Linux is designed to be immutable and always runs from a SquashFS image. This means that the operating system is read-only and cannot be modified at runtime. This immutability provides a strong security posture and means that there is no need to worry about unintended changes to the operating system.
Minimal: Talos Linux is a minimal operating system that only includes the components necessary to run Kubernetes. All the OS is built from the ground-up and no unnecessary components are included. This minimalism reduces the attack surface and improves performance.
Ephemeral and Declarative: Talos Linux nodes are ephemeral and everything written to disk is reconstructible. It is also declarative, meaning that the desired state of the system is defined in a configuration file and gRPC API, which is perfect for someone that loves automation and reproducibility, like myself.
Secure: As a consequence of its design, Talos Linux provides enhanced security features, ensuring that the system remains robust against various threats.
Agnostic: Talos Linux is cloud-agnostic, allowing it to run on various cloud providers and on-premises environments without vendor lock-in.

In summary, and I'm quoting the official documentation, "Talos is meant to do one thing: maintain a Kubernetes cluster, and it does this very, very well."

My thoughts

I have been following the Talos Linux project for a while, and I was gladly surprised to see a Swiss-bank like PostFinance being on the forefront of adopting such modern solutions like Talos Linux and Cluster API.

I think Talos Linux will be a key player in the Kubernetes ecosystem, especially for organizations looking for an on-premises solution that's secure, efficient and easy to manage.

The fact that Talos is declarative and immutable might seem like a drawback at first for someone used to the old ways of managing infrastructure with Ansible or Puppet, but I believe that this is the future of managing Kubernetes clusters.

I want my nodes to behave like pods that I can easily create, destroy, and replace. Besides, I don't want to deal with the overhead of managing the operating system. I already have enough to deal with the on-premises infrastructure for the network and storage and the Kubernetes cluster itself, so why not offload the management of the operating system to a purpose-built distribution like Talos?

With Omni, Sidero Lab's SaaS platform for managing Talos Linux clusters, I believe Sidero Labs have a good revenue model to continue developing Talos Linux. As a fan of open-source, we are all aware of the challenges of maintaining a project like Talos Linux, and I believe that having a SaaS platform to manage Talos Linux clusters is a good way to ensure the project's sustainability.

Talos Linux vs. other solutions

Red Hat OpenShift is a well-known solution in large enterprises. However, more than a Kubernetes distribution, it is a complete platform that includes a lot of features and components, including CI/CD tools, monitoring, etc. It is also expensive.

On the other hand, Talos Linux shines with its simplicity and minimalism, which brings more flexibility and allows teams to choose their solution to complete the platform as they see fit.

RKE2 is another Kubernetes distribution that focuses on simplicity and security, making it a strong contender for organizations looking for a lightweight solution. However, it still requires an underlying operating system that you need to operate.

Bonus

While at KubeCon, I had the opportunity to visit the Sidero Labs' booth and talk to the team behind Talos Linux. I thank the team for a warm welcome and great conversations about the project.

Go further

I wanted to keep this blog post short and not too technical, but if you want to learn more about Talos Linux, I recommend checking out the following resources:

Unveiling the Simplicity of Cluster Mesh for Kubernetes Deployments

Federico Sismondi — Tue, 16 Apr 2024 13:30:57 +0000

Unveiling the Simplicity of Cluster Mesh for Kubernetes Deployments

During Kubecon EU 2024, among a crowd of tech enthusiasts and Kubernetes aficionados, Liz Rice the Queen bee, demo’ed multi-cluster networking. This is Cluster Mesh 101 with Cilium.

Here are a few paragraphs summarizing the experience.

Overview

Cluster Mesh extends the networking plane across multiple clusters. It enables connection among endpoints of connected clusters. Two noticeable features are:
i) Network Policy Enforcement as implemented by Cilium prevails even under this network setup
ii) Services can load balance requests among clusters just by using annotations

Networking Adventures Begin

To the surprise of many, Liz announces she will be running the demo over the venue's Wi-Fi.

Presentation gets started with connectivity tests over VPN connections, routes propagation validation, checks of the BGP peering and visualization of routing tables. The setup is a running k8s cluster in GKE and another in EKS. Node to node Network connectivity is the final objective here, and do not forget all assigned IPs should be not overlapping. Cilium cannot create a bridge between two cloud providers. No black magic.

This is foreplay preparing the ground for the demo. The steps for reproducing this can be found in official Cilium here.

Enter Cilium's Cluster Mesh

With the foundational work laid, it's time to kickstart the demo. Liz demonstrates how to enable all necessary components with cilium clustermesh enable in both clusters. These trigger the deployment of the clustermesh-apiserver into each cluster, along with the generation of all required certificates. This component goes the extra mile by attempting to auto-detect the optimal service type for LoadBalancer, ensuring the Cluster Mesh control plane is efficiently exposed to other clusters.

A simple cilium clustermesh connect builds the bridge between clusters, just as if we had a single network plane between Pods across all clusters.

Now cilium clustermesh status echos:

✅ All 2 nodes are connected to all clusters
🔌 Cluster Connections:

cilium-cli-ci-multicluster-2-168: 2/2 configured, 2/2 connected

Traffic load balancing and failover in the hive:

Till now demo showed and validated the piping between Pods. We can successfully communicate with a Pod on another cluster using its IP address for example. But this wouldn't be very practical in a real world-scenario. Moreover no DNS service can help us fetch this dynamic IP. In fact, this is the raison d'être of k8s Service object.

What Cilium proposes is extending native Service resources into a cluster-mesh-aware Service using annotations.

Cilium provides a pragmatic solution through global services with auto discovery and failover mechanisms.

By using service.cilium.io/global=true makes a service global, which meansmatching Pods across clusters. Or in other words we extend service’s backends to use Pods in remote clusters. Then the service’s traffic is balanced across clusters.

Questions? This is probably better explained by Ciilium documentation here

apiVersion: v1
kind: Service
metadata:
  name: rebel-base
  annotations:
    service.cilium.io/global: "true"
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    name: rebel-base

With service.cilium.io/affinity=local|remote we can fine tune the global service to prefer local or remote Pods. With local we could designate our local cluster as the primary destination, while the remote Pods serve as a backup.

The following represents a service, which is global and prefers using endpoints found locally:

apiVersion: v1
kind: Service
metadata:
  name: rebel-base
  annotations:
    service.cilium.io/global: "true"
    service.cilium.io/affinity: "local"
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    name: rebel-base

Should the primary cluster encounter any issues or downtime, traffic seamlessly shifts to the backup cluster, ensuring continuity of service.
In essence, Cilium offers a straightforward approach to traffic management, enhancing reliability by providing a failover mechanism that ensures service accessibility remains intact in the face of pod disruptions.

Conclusion

A few interesting use cases arise when using Cluster Mesh. The one we focused on in this article is about leveraging remote clusters Pods as service's Pod backend as failover mechanism. We can also think of using a global service for moving workloads around, for example for lowering computational costs into cheaper regions.

So there you have it, folks. A whirlwind tour of multicluster networking traffic management with Cilium, served up with a dose of honey.
Who knew cluster meshing would be that simple?

Contact us

Needs a demo, or dig into some specifics on Cilium?
Ping us here: https://camptocamp.com/consulting

Beyond the Buzz: Embracing the Magic of eBPF in Kubernetes

Julien Acroute — Wed, 10 Apr 2024 14:02:32 +0000

Beyond the Buzz: Embracing the Magic of eBPF in Kubernetes

In a time where the buzz around Artificial Intelligence (AI) seems to overshadow everything else, this year's KubeCon Europe offered a refreshing perspective. While AI continues to be a hot topic, some in the Kubernetes community are starting to feel a bit tired of it. With all the hype and uncertainty surrounding AI, another hero has emerged: eBPF (Extended Berkeley Packet Filter).

AI: A Distant Shining Horizon

AI has certainly added some excitement to discussions about cloud-native technologies, from automating cluster troubleshooting to hosting AI on Kubernetes. But not everyone is fully on board. While AI has proved to be of great assistance - e.g. suggesting, fixing and reviewing code written by humans- some folks worry that too much focus on AI might distract from more practical, ready-to-implement advancements. The feeling is clear: while AI offers lots of possibilities, the horizon is a bit uncertain and overly hyped.

eBPF: Here and Now in Kubernetes Innovation

In contrast, eBPF is all about practical innovation. It's a technology that delivers real results, right here, right now. We need solutions for network security, observability, and performance today, and that's where eBPF shines. Unlike the abstract promises of AI, eBPF offers concrete tools and methods to improve Kubernetes environments immediately. For example, when it comes to network security, eBPF-powered tools like Cilium can do the job without needing the complexity of AI.
This shift towards valuing what's immediately useful over what's exciting but distant was noticeable at KubeCon. As we dive deeper into what eBPF can do, it becomes clear why this technology has captured the attention of the Kubernetes community.

The Way Forward with eBPF

By embracing eBPF, the Kubernetes community isn't just adopting new tools; it's championing a philosophy of practical, tangible progress. As we explore the latest eBPF innovations – from better security to revolutionary observability tools – we see the benefits eBPF brings to Kubernetes. It's a journey grounded in reality, offering not just a vision of the future, but a roadmap to get there.

Redefining Efficiency: eBPF's Radical Return to Linux's Roots

eBPF represents a refreshing departure from the conventional approach of stacking layer upon layer in pursuit of functionality, often resulting in bloated, resource-intensive systems, even for simple tasks like rendering a webpage or routing network packets between nodes or clusters. With eBPF, we're venturing back into the depths of the Linux system, where innovation meets efficiency. Here, we witness a paradigm shift—a departure from the status quo. The results speak for themselves: a nearly negligible overhead and remarkable responsiveness. In fact, eBPF brings us so close to real-time processing that it's revolutionizing how we think about performance in cloud-native environments. We're not just optimizing; we're redefining what's possible, and eBPF is leading the charge.

eBPF talks from KubeCon 2024

Cilium: Connecting, Observing, and Securing Service Mesh and Beyond with eBPF
Speakers: Liz Rice, Christine Kim, Nico Meisenzahl, Vlad Ungureanu
https://youtu.be/wq1TxZw1AaY?si=JTyhE333QfsGht0T
Dealing with eBPF’s Observability Data Deluge
Speaker: Anna Kapuścińska
https://youtu.be/yWB8n_e4N14?si=OyMJEKzbxS5zxA5P
Unlock Energy Consumption in the Cloud with eBPF
Speaker: Leonard Pahlke
https://youtu.be/lW9pZoKRJVs?si=rX5CQMaFuZBm8bBT
Fast and Efficient Log Processing with Wasm and eBPF
Speaker: Michael Yuan
https://youtu.be/4u7nUpZxr3g?si=pkcpoEwOeDaH5HcE
No 'Soup' for You! Enforcing Network Policies for Host Processes via eBPF
Speaker: Vinay Kulkarni
https://youtu.be/AWAf3H4Qwq8?si=qVqQfWb3J_905BCJ
eBPF’s Abilities and Limitations: The Truth
Speakers: Liz Rice & John Fastabend
https://youtu.be/tClsqnZMN6I?si=TyMFTMk4Q45K6T2v

Use Tetragon to Limit Network Usage for a set of Binary

Julien Acroute — Thu, 03 Aug 2023 07:39:02 +0000

A matter of trust

Many interesting software are coming from the community, many are distributed through the package manager of the operating system. But for the others, you can download them from Github release pages, use snap or homebrew to cite a few. But this last installation method bypasses the security team that tries to improve the security of your operating system. By doing so, you are implicitly trusting the author he is not distributing malware or implementing backdoors. How many tools did you install by hand? Do you really trust all of them? Confidence is very important, yet it would be nice to limit capabilities for a set of binary that you don't fully trust. In this blog post, we will use Tetragon to forbid network usage for tools that don't need to.

Goal

We will separate tools installed locally into two families. On one side, tools that need network access in a specific directory. Another directory for tools that don't need internet access.

For example the following tools use network sockets:

while these do not:

We will move the last tools in a specific directory ~/bin-no-network/ and use Tetragon to inject a policy in the kernel as a eBPF program to kill any binary located in ~/bin-no-network/ trying to open a network socket.

Tetragon installation

Tetragon is "Kubernetes-aware" but it can also be used outside Kubernetes on a regular workstation. You can deploy Tetragon as a container:

$ docker run --name tetragon --rm -d                 \
    --pid=host --cgroupns=host --privileged          \
    -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
    quay.io/cilium/tetragon:v0.10.0

Because Tetragon needs to inject code in the kernel, we need to bypass most of the docker isolation mechanism. We only use the packaging feature of docker to avoid installation of system libraries and the binary. So you will have to trust ;-) this tetragon binary because it will run like a process running as root on your workstation. Note the mount point: /sys/kernel/btf/vmlinux, this is a kind of bridge to eBPF kernel features.

eBPF Principle

When using eBPF, applications are always composed of two parts, one deployed in the Kernel as an eBPF program and another running as a regular program in "user space". The user space program (Tetragon) will inject a small eBPF program in the kernel to intercept some system calls. This means that every application that uses this system call will trigger this eBPF program that can observe or modify the system call result. A specific data structure is then created in the kernel: a ring buffer. This is used by the eBPF program to store some interesting data. Finally, the user space program Tetragon, which has also read-only access to this data structure, will be able to retrieve information gathered by the eBPF program.
One good point with this architecture is that evaluation of rules is done within the kernel and does not require communication with the user space program. The only drawback is that information observed by the eBPF program can be overridden by new incoming information if the user space part does not read information fast enough. This is how ring buffers are designed.

Writing a Policy

Our goal is to forbid network usage coming from binaries in a specific folder.
For this we will need a CLI to interact with tetragon. We can use the tetra binary in the docker container:

$ alias tetra="docker exec -ti tetragon tetra"
$ tetra version
server version: v0.10.0
cli version: v0.10.0

Even if we are not using Kubernetes, we still need to write some yaml as Tetragon only understands TracingPolicy objects. A TracingPolicy is a kubernetes custom resource to install hooks in the kernel and actions.

Let's start with a TracingPolicy from the Tetragon documentation:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int

This Tracing policy will just observe network events:

socket creation (tcp_connect)
traffic in the socket (tcp_sendmsg)
socket close (tcp_close)

Now we need to add:

an action when this kind of event is detected
a filter to only apply this action if this is generated from a binary located in our specific directory ~/bin-no-network/.

We need to add a selectors section in the TracingPolicy and use the matchBinaries selector to apply this policy only for binary in the ~/bin-no-network/ folder:

    selectors:
    - matchBinaries:
      - operator: "In"
        values:
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"

Unfortunately, only 'In' and 'NotIn' operator are implemented for the matchBinaries selector. We cannot use the 'Prefix' operator like this:

    selectors:
    - matchBinaries:
      - operator: "Prefix"
        values:
          - "/home/jacroute/bin-no-network/"

So the TracingPolicy should look like:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: &selector
    - matchBinaries:
      - operator: "In"
        values:
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"
      matchActions:
      - action: Sigkill
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: *selector
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int
    selectors: *selector

Deploy the TracingPolicy

We first need to transfer the file to the container. Suppose that you stored the TracingPolicy in bin-no-network.yaml file, you can transfer the policy using the docker cp command:

$ docker cp bin-no-network.yaml tetragon:/tmp/bin-no-network.yaml

Then we can use the tetra cli to deploy this policy:

tetra tracingpolicy add /tmp/bin-no-network.yaml

Testing the TracingPolicy

Now we can test if the policy is blocking network access for the two binaries listed in the policy.

$ mkdir ~/bin-no-network/
$ cp /usr/bin/curl ~/bin-no-network/
$ cp /usr/bin/jq ~/bin-no-network/
$ ~/bin-no-network/curl google.fr
[1] 122677 killed ~/bin-no-network/curl google.fr
$ echo "{}" | ~/bin-no-network/jq
{}

curl tried to open a socket but the process was killed. jq does not try this kind of system call and was not killed.

Automatically populating the Policy

We can build a shell script to maintain the list of binaries synchronized with the content of the ~/bin-no-network/ directory. Using find we can list binaries in this folder:

$ find ~/bin-no-network/ -executable -type f
/home/jacroute/bin-no-network/curl
/home/jacroute/bin-no-network/jq

Then with awk with can add surrounding spaces and quotes needed to integrate the yaml:

find ~/bin-no-network/ -executable -type f | awk '{ print "          - \""$0"\""}'
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"

Finally, integrate this in the yaml and inject the policy in the kernel with the following script:

#!/bin/bash

docker exec tetragon tetra sensors rm connect

policy=$(mktemp -p /dev/shm)

cat << EOF > $policy
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: &selector
    - matchBinaries:
      - operator: "In"
        values:
$(find ~/bin-no-network/ -executable -type f | awk '{ print "          - \""$0"\""}')
      matchActions:
      - action: Sigkill
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: *selector
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int
    selectors: *selector
EOF

docker cp $policy tetragon:/tmp/policy.yaml
docker exec tetragon tetra tracingpolicy add /tmp/policy.yaml
rm $policy

Building trust in your relationship with your workstation XD

After a first date with Tetragon during the Kubernetes Community Days during 2023, I was seduced by the way it's implemented: most of the time new features are implemented on top of others creating a complex multi layered cathedral. Perfect for a wedding, but I prefer simplicity.

Using eBPF to enforce "security" seems to be very efficient from the performance point of view. Bypassing Tetragon security by denial of service attacks is unlikely. For example, the sigkill action triggered by the Tetragon’s eBPF program is done synchronously, in the kernel and not at user space, which makes it hard to bypass. In other words, the security mechanism is fully and autonomously implemented at kernel level.

This mechanism proves to be effective in blocking network activity and building trust in our beloved workstation even when running potentially malicious binaries.

Using Tetragon in a Kubernetes context is the next step. The plan is to observe the behavior of an application in a development environment (files read/write, command executed) and generate a profile. Then, promote this profile from development to production, and enforce an allow-only behavior using Tetragon.

Using ArgoCD Pull Request Generator to review application modifications

Julien Acroute — Tue, 11 Apr 2023 14:29:12 +0000

As a developer, when modifications are pushed to a feature branch, you and your team want to test this new feature. If you have the chance to work with a stateless application, you can deploy another instance of the application with modifications from the feature branch.

An interesting feature of ArgoCD is the Pull Request Generator. It's a generator for ApplicationSet. An ApplicationSet is a template of ArgoCD Application associated with a generator. Generator can be a directory: an application will be created for every sub-folder. There is also the Cluster generator that deploy the same Application but in every cluster managed by ArgoCD.

The syntax for the Pull Request generator is quite simple:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapps
spec:
  generators:
  - pullRequest:
      github:
        owner: camptocamp
        repo: myrepository
        tokenRef:
          secretName: github-token
          key: token
        labels:
        - deploy
  template:
    ...

Before testing this feature what is expected ?

A new ArgoCD Application is created when a pull request is created with a "deploy" label
In the template of the Application we can use metadata of the pull request: ID, title, description, labels, source and target branch name, commit ref, …
A comment is added to the PR when the Application is deployed and Synced with the available URLs

Now we will see how to implement this ;-)

Workflow before Implementation

Source Code Repository

Let's start with a simple application repository: git@github.com:camptocamp/frontend.git
Let's consider that this repository has a github action that builds a container image when a pull request is opened. The image is tagged with the short commit hash and the concatenation of the branch name and the short commit hash. For example, if the feature branch is name update_lib with the last commit: 4a9b29e, the following tags will be generated:

4a9b29e
update_lib-4a9b29e

Deployment Repository

There is another git repository: git@github.com:camptocamp/argocd-project-foo-apps.git to describe what needs to be deployed in each kubernetes cluster (dev, int, …). In this repository, we have one folder per environment:

apps
├── dev
│   ├── backend
│   └── frontend
├── int
│   ├── backend
│   └── frontend
└── prod
    ├── backend
    └── frontend

We are using an ApplicationSet to deploy every component defined in this git repository using the directory generator. For example, for "dev" env:

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: foo-dev
  namespace: argocd
spec:
  generators:
  - git:
      directories:
      - path: apps/dev/*
      repoURL: git@github.com:camptocamp/argocd-project-foo-apps.git
      revision: main
  template:
    metadata:
      name: foo-dev-{{path.basename}} # sub folder name
    spec:
      source:
        path: '{{path}}' # full path
        repoURL: git@github.com:camptocamp/argocd-project-foo-apps.git
        targetRevision: main
      project: default
      destination:
        namespace: foo-dev
        server: https://kubernetes.default.svc

This ApplicationSet will create one ArgoCD Application per folder in apps/dev/. Imagine that we have the following structure in git :

apps
└── dev
    ├── backend
    │   ├── Chart.yaml
    │   └── values.yaml
    ├── database
    │   ├── Chart.yaml
    │   └── values.yaml
    └── frontend
        ├── Chart.yaml
        └── values.yaml

This will generate 3 Argocd Applications :

foo-dev-database
foo-dev-backend
foo-dev-frontend

Review App Workflow

When a pull request is opened for a specific application, for example the frontend, we want to deploy a new instance of this specific component. We will monitor pull requests on the frontend repository.
So we want to create an additional ApplicationSet to deploy the frontend if a pull request is created with the label deploy in the frontend git repository.

Implementation

Create a token to access GitHub API

First step is to create a token to access the frontend repository.

Then we need to deploy this token as a kubernetes secret in the cluster :

export GITHUB_TOKEN=ghp_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
kubectl create secret generic github-token \
  --from-literal=token=$GITHUB_TOKEN \
  -o yaml --dry-run=client > github-token-secret.yaml

If you follow the GitOps principles to deploy manifests in the cluster, you can commit this file.

Create a webhook

For a better user experience, we can setup a webhook that will notify ArgoCD when something changes on Github.
This webhook needs to be defined in the source code repository, where pull requests are created.
Go to the repository "Settings" and then "Webhooks". Click on the "Add webhook" button.

The Payload URL is the URL to access ArgoCD with the path /api/webhook.
The Content Type should be set to application/json.
It's a good idea to set a "Secret", just use a random string, we will use this random string in the next step.
Regarding events, we need to individually selects events and choose "Pull requests" events.

Deploy the new ApplicationSet

This new ApplicationSet will use the "Pull request" generator, this "generator" will monitor pull requests on the source code repository. Just create and commit a file with the following ApplicationSet manifest and deploy this new manifest.

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: foo-dev-review-frontend
  namespace: argocd
spec:
  generators:
  - pullRequest:
      github:
        owner: camptocamp
        repo: frontend # This is the application source code repo
        tokenRef:
          secretName: github-token
          key: token
        labels:
        - deploy # label on PR that trigger review app
  template:
    metadata:
      name: 'foo-dev-frontend-{{branch}}-{{number}}'
    spec:
      source:
        repoURL: git@github.com:camptocamp/argocd-project-foo-apps.git
        targetRevision: main
        path: apps/dev/frontend
        helm:
          parameters:
          - name: "image.tag"
            value: "{{branch}}-{{head_sha}}" # override of the image tag
          - name: "ingress.prefix"
            value: "{{branch}}" # add a prefix to the URL 
      project: default
      destination:
        server: https://kubernetes.default.svc
        namespace: foo-dev

This ApplicationSet will deploy a new instance of the frontend for each pull request with the deploy label. The image tag will be overridden with the branch name and the short commit hash and a unique prefix will be added to the URL to avoid conflicts with other instances. Finally, the name of the release is also unique as it's the same as the Application name: foo-dev-frontend-{{branch}}-{{number}}, this should also avoid conflicts on object names.

Configure ArgoCD to use secret for webhook

The last step is to protect the ArgoCD webhook with a password. The ArgoCD Helm chart allows setting a secret for the github webhook endpoint : configs.secret.githubSecret.

Testing

Now it's time to test. For this step, you just need to make a modification in the source repository and create a pull request for this. Please wait until the container image is built. Then by adding the deploy label a new ArgoCD app should be created :-)

Conclusion

This new ArgoCD feature is very interesting. Maybe it can help to have feedback on the pull requests to see the status of the review app. It can also be interesting to have the list of URLs available. ArgoCD already shows this information in the webUI. So maybe just a link to the ArgocD application is enough.

Camptocamp will participate in the KubeCon at Amsterdam from 18 to 21 april. Maybe we can meet there and discuss development workflows that really help developers.

Deploy your Pulumi project using Docker and Dagger.io

Hugo Bollon — Wed, 14 Dec 2022 10:30:00 +0000

🕰️ In the previous episode

In the first part of this Dagger's series, I showed you what's Dagger.io, what's the features of it and it's benefits against others ci/cd solutions and finally the very basis of Dagger.

With this chapter, I will show you how we can overpower the CI/CD of any Pulumi project using Dagger.

🧰 Pulumi - An amazing IaC tool

First of all, I think that some of you may doesn't know what is Pulumi or even IaC (Infrastructure as Code) concept, so I will quickly present to you these two points.

Infrastructure as Code

Nowadays, IT extends to many areas, and so new needs are emerging leadingly to a necessity to adapt infrastructures in order to be able to support all of this.
They also have seen their prerequisites evolve given their multiplication and their increasingly large sizes. As a result, companies started wanting to automate and simplify their infrastructures.

To address this problem, Amazon unveiled in 2006 the concept of Infrastructure As Code (or IaC) allowing, on Amazon Web Services, the configuration of instances using computer code.
It was a revolution for infrastructure management and, although limited at the time, this method was quickly adopted by the market.
This event also coincides with the date of appearance of the DevOps movement in which it's part.

Most of the infrastructure as code tools are based on the use of descriptor files to organize the code which avoids duplication between environments. Some advanced tools support variability, the use of outputs and even deployment to several providers simultaneously.

There are three types of infrastructure as code:

Imperative: resources (instances, networks, etc.) are declared via a list of instructions in a defined order to obtain an expected result.
Functional: unlike the imperative mode, the order of the instructions does not matter. The resources are defined in such a way that their final configuration is as expected.
Based on the environment: the resources are declared in such a way that their state and their final configuration are consistent with the rest of the environment.

The advantages of IaC compared to traditional management are numerous, such as: cost reduction, the possibility of versioning the infrastructure, the speed of deployment and execution, the ability to collaborate, etc.
It also allows complete automation, in fact, once the process is launched, there is no longer any need for human intervention. This advantage not only limits the risks due to human error and therefore increases reliability, but also allows teams to focus on projects and less on the deployment of applications.

Pulumi

Pulumi is an open-source IaC tool. It can be used to create, manage and deploy an infrastructure on many cloud provider like AWS, GCP, Scaleway, etc.
It haves few strengths compared to others IaC solutions like Terraform for example.

First of all, Pulumi support many programming languages like Go, TypeScript, Python, Java, F# and few others.
This is a great advantage for Pulumi and one of the reasons why we begin to use it at Camptocamp because the presence of a programming language like Go, rather than a configuration language like HCL (language used by Terraform) allows much more flexibility and adaptability.

Furthermore, Pulumi has new native providers like AWS, Google, and others in order to get new versions the same day they are released.
Additionally, Pulumi also supports Terraform providers to maintain compatibility with any infrastructure built using Terraform.

Another very interesting advantage is the support of what they call “Dynamic Providers” which allows to easily extend an existing provider with new types of personalized resources and that by directly programming new CRUD operations.
This can allow new resources to be added, for example, while adding complex migration or configuration logic.

Finally, there are still many advantages to Pulumi such as the ease of carrying out tests thanks to the native frameworks of the programming languages provided for this usage, the presence of aliases allowing a resource to be renamed while maintaining compatibility with the state of the infrastructure, better integrations with mainstream IDEs like VSCode, etc.

⚡ Supercharge your Pulumi project thanks to Dagger

Now that you know IaC, Pulumi and of course Dagger (if not, you can check the first part of this blog series), we will see how we can create CI/CD pipelines for any Pulumi project using Dagger and CUE and finally how can we run them.
For that, I will present to you the Dagger architecture we have built at Camptocamp, it was designed to be complete and reusable. It may be too complex for small projects but if you understand it you will normally be able to create your own !

Important note: Initially Dagger.io was developed using Cuelang, an amazing and modern declarative language. Cue was also the only way to do pipelines with it. However, Dagger's team have recently transformed Dagger to be language agnostic.
We now have different SDKs: Go (the main one), CUE, Node.js and Python.
As of today, I advice you to choose the Go SDK if you don't really know which one to take. For the CUE one, I only recommend it to you if:

You like declarative language like YAML
You know CUE or want to learn it

For the following chapters, I will present our implementation which is made in CUE. However, it will be easy to transpose it to other SDKs if you understand it.

📦 The Pulumi package

In order to make a reusable and powerful Dagger project, we decided to create a "Pulumi" package which embeds our objects definitions like:

the Docker image
the container definition
the command object

As an exemple there is the command object definition:

// pulumi/command.cue
package pulumi

import (
    "dagger.io/dagger"
    "universe.dagger.io/bash"
    "universe.dagger.io/docker"
)

#Command: self = {
    image?: docker.#Image
    name:   string
    args: [...string]
    env: [string]: string
    source: dagger.#FS
    input?: dagger.#FS

    _forceRun: bash.#RunSimple & {
        script: contents: "true"
        always: true
    }

    _container: #Container & {
        if self.image != _|_ {
            image: self.image
        }

        source: self.source

        command: {
            name: self.name
            args: self.args
        }

        env: self.env & {
            FORCE_RUN_HACK: "\(_forceRun.success)"
        }

        if self.input != _|_ {
            mounts: input: {
                type:     "fs"
                dest:     "/input"
                contents: self.input
            }
        }

        export: directories: "/output": _
    }

    output: _container.export.directories."/output"
}

At the top of this file, you can see the package definition and all our imports. universe.dagger.io is a community repository where we can find pre-made package for Docker, bash, alpine, etc.
Just after that, we start defining our #Command object by adding public fields like command's name, args, custom Docker image (which is optional due to ? character, etc.
We also define our unexported fields (which aren't accessible from outside the package) like the container definition which will run our command (the container object is defined in the container.cue file in the same repository).

This is one of the few definitions that we have in this package, I will not show you directly all of them since it's really specific to our implementation and it's not very relevant to explain how it's working.
However, you can retrieve all of our source code here: https://github.com/camptocamp/dagger-pulumi

So one last file from this pulumi package that we will see is the main.cue one. It's where we defined all the Pulumi commands that we will use (using the #Command object seen just before).

// pulumi/main.cue
[...]
#Preview: self = {
    stack: string
    diff:  bool | *false

    #Command & {
        name: "preview"

        args: [
            "--stack",
            stack,
            "--save-plan",
            "/output/plan.json",

            if diff {
                "--diff"
            },
        ]
    }

    _file: core.#ReadFile & {
        input: self.output
        path:  "plan.json"
    }

    plan: _file.contents
}

#Update: {
    stack: string
    diff:  bool | *false
    plan:  string

    _file: core.#WriteFile & {
        input:    dagger.#Scratch
        path:     "plan.json"
        contents: planHere, you defined 
    }

    #Command & {
        name: "update"

        args: [
            "--stack",
            stack,
                        [...]
            if diff {
                "--diff"
            },

            "--skip-preview",
        ]

        input: _file.output
    }
}
[...]

I voluntary remove some parts of the file and leave only the definition for Pulumi preview and update commands in order to keep it simple.
In the Pulumi context, preview command is used to compare the codebase which defines the desired infrastructure state with the actual one and preview all the potential changes if we apply the configuration.
The update one, as it name says, update the deployed infrastructure to match the desired state.

Using these definitions, we finally defined a #Pulumi object in the main.cue from the parent ci package.
It's composed by few attributes used to set our project environment like, for example, the Pulumi stack (workspace), the env variables, the authorization to run destructive actions through this CI, etc.
We also have a list of all available commands construct using all #Command objects defined earlier.

Finally, the last step before we can be able to run our pipelines is to create the Dagger's plan.
This is the most important part where all jobs are defined but it's really specific to the project for which it's built.
I will present to you a plan from one of our projects where we used the pulumi package that I show you just before.

import (
    "github.com/camptocamp/pulumi-aws-schweizmobil/ci"

    "dagger.io/dagger"
)

dagger.#Plan & {
    client: {
        env: {
            PULUMI_GOMAXPROCS?: string
            PULUMI_GOMEMLIMIT?: string
        }

        filesystem: {
            sources: read: {
                path:     "."
                contents: dagger.#FS

                exclude: [
                    ".*",
                ]
            }

            // TODO
            // Simplify once https://github.com/dagger/dagger/issues/2909 is fixed
            plan: read: {
                path:     "plan.json"
                contents: string
            }

            planWrite: write: {
                path:     "plan.json"
                contents: actions.preview.plan
            }
        }
    }

    #Pulumi: ci.#Pulumi & {
        env: {
            if client.env.PULUMI_GOMAXPROCS != _|_ {
                GOMAXPROCS: client.env.PULUMI_GOMAXPROCS
            }

            if client.env.PULUMI_GOMEMLIMIT != _|_ {
                GOMEMLIMIT: client.env.PULUMI_GOMEMLIMIT
            }
        }

        source: client.filesystem.sources.read.contents
        stack:  string
        diff:   bool
        update: plan: client.filesystem.plan.read.contents
        enableDestructiveActions: bool
    }

    actions: {
        #Pulumi.commands
    }
}

Firstly, you can see at the begin of this file the import of the homemade ci package (as a reminder, it is fully available here)
After, we defined the dagger.#Plan object which is composed by few parts:

the client config: this is where we will be able to set some runtime elements like: env variables, filesystems with read/write permissions, the sources location...
the actions: this is all the possible jobs that you will be able to run using Dagger CLI or in your CI/CD environment, in our case we just use local variable #Pulumi.commands which is built using the #Pulumi object defined in the ci package.

🚀 Launch Dagger's jobs

Locally using CLI

As I told in the previous part of this blog series, Dagger can run any jobs locally thanks to his Dockerize design.
You must have installed the Dagger's CLI for Cue SDK (check the first part of this series or the official documentation if it's not already done).

Once ready, you can open your favorite terminal client located inside your project directory (composed by your Pulumi project and your Dagger plan) and run:

dagger --plan=ci.cue do --with '#Pulumi: stack: "<your_stack>"' preview -> it will run a Pulumi preview on the desired stack
dagger --plan=ci.cue do --with '#Pulumi: stack: "<your_stack>", #Pulumi: diff: true, #Pulumi: enableDestructiveActions: true' update -> it will run a Pulumi update on the desired stack

Remotely using a CI/CD environment

With Dagger you can also run your pipelines on every CI/CD environments!
As a simple example, you can easily run these jobs on Github Actions:

name: pulumi-project

on:
  push:
    branches:
      - main

jobs:
  dagger:
    runs-on: ubuntu-latest
    steps:
      - name: Clone repository
        uses: actions/checkout@v2

      - name: Install Dagger Engine
        run: |
            cd /usr/local
            wget -O - https://dl.dagger.io/dagger/install.sh | sudo sh
            cd -

      - name: Run Pulumi update using Dagger
        run: |
            dagger-cue --plan=ci.cue do update --log-format plain

Note than the --log-format plain flag is useful in order to have well formatted logs in the Github Action output.
With this action, a pulumi update will be launched at every push on the main branch using Dagger.

🔚 Conclusion

If you have made it this far I thank you very much and I hope I have succeeded to give you a good overview of what is Dagger and how is it possible to exploit it in order to improve its practices in terms of CI/CD.

From my point of view, Dagger is a very promising solution. Right now we can still feel Dagger's young age, indeed, it's still not perfect.
In my opinion, each SDK should continue to develop and harmonize with each other. I also think that Dagger's execution performance and cache management should be improved (I'm basing myself on the 0.2 version of the engine since 0.3 is not yet available with the Cuelang SDK so maybe that's already better!).
Nevertheless, the advantages of Dagger are already essential for me, such as the possibility of running its pipelines locally or even being able to run them remotely on a remote VM using local sources.

To be followed very closely! 😉

Use Docker to build better CI/CD pipelines with Dagger

Hugo Bollon — Sun, 11 Dec 2022 05:00:00 +0000

With the raises of DevOps practices, CI/CD (continuous integration & continuous deployment) takes a major place in every delivery workload.
CI/CD allow organizations to build, test and finally ship their applications more quickly and efficiently. It's a modern set of practices which allows to automatically trigger build, test or others types of jobs when the changes to the codebase are done.

In this quest of automation, we can use some CI/CD ecosystem like Github Actions, Gitlab-CI or many more.
However, a very promising new solution open-source is born called Dagger.

🤔 Dagger? What is it?

Dagger.io is a brand-new programmable CI/CD engine which is open-source. It was created by Solomon Hykes, the founder of Docker.
It's designed to use Buildkit from Docker in order to runs our pipelines inside containers.
For those who don't know, Buildkit is an improved backend used by Docker to build images. It's way more efficient than the old legacy Docker's builder due to its improved caching system, the parallelization of build tasks and the support of new features in Dockerfile.

Dagger is also programmable, so we must create our pipelines as code, Dagger himself is written in CUE (Cuelang), a Google's langage.

CUE is an acronym of "Configure Unify Execute" and so as its name suggests, it's not another general purpose language but a declarative one mainly used for data templating and validation, configuration or even code generation.

It's basically an evolution of more lambda languages like YAML or JSON and one of the thing which make it way better and modern is the presence of a package manager.

So going back to Dagger, you can use CUE to build your pipelines but you can also use the SDK available with multiples languages support:

Go
Python
NodeJS

✨ Features and advantages

Dagger, thanks to his innovative containerized design and architecture, leads to many advantages over more conventional CI/CD methods.
As seen previously, Dagger works with Docker containers allowing it to be cross-compatible with every CI/CD runtime environment like Github Actions, Gitlab-CI, Travis-CI, etc.

One of the other main strengths of Dagger is the ability to do local testing of our pipeline using the Dagger CLI. This is made possible by the Dockerized design of it which makes the development and testing processes way easier compared to conventional CI/CD solutions.
Conversely, Dagger is also able to perform remote runs (directly on a self-hosted Github runner for example) with local sources.
To do this it is possible to use the environment variable DOCKER_HOST.

The Dockerized design also allows pipelines made with Dagger devkit to be run in every CI/CD runtime environment like, for example, Github Action (using the official Dagger Github Action from the marketplace).
Furthermore, it can also be run independently of the architecture of the platform. The only requirement is the Docker ecosystem support. So it can be run on a managed runner (eg. Github Runners), a self-hosted runner, a local machine, a serverless compute instance, etc.

Furthermore, Dagger has a solid caching system which caches every operation by default, but it's customizable.
It helps to reduce execution time of CI/CD jobs after the first runs by caching some unchanged required files like: the downloaded dependencies, some built binaries or just some CI/CD engine stuff.

Finally, Dagger is designed to be reusable thanks to his internal package manager (provided by the Cue language). Indeed, it's similar to the one of the language Golang.

package app

import (
    "dagger.io/dagger"
    "dagger.io/dagger/core"
    "universe.dagger.io/bash"
    "universe.dagger.io/docker"
)

Here you can see the import of some packages from the "universe". The Universe is a community package repository, all sources of these packages can be found here, you can also contribute to them.

❓ How does it works?

Dagger is language agnostic, the wish of the team behind is to allow developers to make their pipelines with the language of their choice.
For that, Dagger use a specific architecture. The SDKs (Go, Cue, Node and Python) don't actually run your pipelines themself. Instead, they send pipeline definitions to the Dagger GraphQL API which will after trigger the Dagger Engine.

Dagger's architecture scheme. Ref: https://dagger.io/blog/graphql

🎮 Demo application using Cue SDK

Dagger's team made a demo app with pipelines designed to execute build and test jobs. It's useful to discover and try Dagger.
To use it, you must, of course, have Docker installed since Dagger will use Buildkit but you must also install the CLI (the CUE one is available here: https://docs.dagger.io/sdk/cue/526369/install)
Once done, clone this repository: https://github.com/dagger/todoapp/blob/main/dagger.cue
Finally, open a terminal inside this freshly cloned project and run dagger-cue project update.

Now you're ready to get your hands in Dagger, if you look at the dagger.cue file you will see the plan.
A plan in Dagger orchestrates the Actions. It's the base component of your configuration.

Within this plan we can:

interact with the client filesystem
- read files, usually the current directory as .
- write files, usually the build output as _build
read and define env variables
declare jobs like dependencies update, build & test

There is the one of this demo app:

package todoapp

import (
    "dagger.io/dagger"

    "dagger.io/dagger/core"
    "universe.dagger.io/netlify"
    "universe.dagger.io/yarn"
)

dagger.#Plan & {
    actions: {
        // Load the todoapp source code 
        source: core.#Source & {
            path: "."
            exclude: [
                "node_modules",
                "build",
                "*.cue",
                "*.md",
                ".git",
            ]
        }

        // Build todoapp
        build: yarn.#Script & {
            name:   "build"
            source: actions.source.output
        }

        // Test todoapp
        test: yarn.#Script & {
            name:   "test"
            source: actions.source.output

            // This environment variable disables watch mode
            // in "react-scripts test".
            // We don't set it for all commands, because it causes warnings
            // to be treated as fatal errors.
            // See https://create-react-app.dev/docs/advanced-configuration
            container: env: CI: "true"
        }

        // Deploy todoapp
        deploy: netlify.#Deploy & {
            contents: actions.build.output
            site:     string | *"dagger-todoapp"
        }
    }
}

You can see that three different jobs are defined:

a build one which will compile the react app using yarn
a test one
a deploy one

To execute locally one of those, you can execute: dagger-cue do <job_name>.
So to build the project you can do dagger-cue do build.

⛏️ To go deeper

I hope to have given you a good overview of Dagger with this first chapter or at least to have made you want to go further with it.
Getting started with Dagger and CUE is not necessarily easy.
It's still an extremely young product (we are currently in version v0.2.36) and there is, therefore, still a lot of optimization work to be done as well as missing features.
Keep in mind that Dagger is an open-source project available on Github so don't hesitate to open issues or pull request if needed.

In the next (coming soon) chapter we will see how to use Dagger to build a deployment workflow for a Pulumi project and how to manage secrets.

How I deployed a serverless and high availability Blackbox Exporter on AWS Fargate

Hugo Bollon — Mon, 04 Jul 2022 14:19:14 +0000

At Camptocamp, we're using multiple Blackbox Exporters hosted in a few different cloud providers and world regions. We're using them to monitor availability and ssl certificate validity and expiration of many websites.
They were all deployed inside Linux VMs provisioned by Terraform and configured by our Puppet infrastructure. However, in order to achieve more simplicity and high availability, we wanted to deploy containers instead of these VMs.

🧐 Why a serverless approach with AWS Fargate

AWS ECS (Elastic Container Service) is a fully managed, highly scalable and docker compatible container orchestration service.
It is widely used to host microservice applications like webservers, APIs or machine learning applications.

With ECS, you're free to choose between EC2 or Fargate instances to run your apps.
Fargate is a serverless compute engine which allows you to just focus on building and deploying your apps by taking away all infrastructure deployments and maintenance. No need to worry about security or operating systems, AWS will handle that.
On the other hand, EC2 is more flexible than Fargate and less expensive. It can also be interesting for some customers to manage the security themselves.

In our case, we opted for a serverless approach using Fargate in order to take advantage of the simplicity of a managed infrastructure since for blackboxes we have no specific security constraints for the infrastructure.

🧳 What I use

To deploy an application on ECS using Fargate you will need three different components:

An ECS Cluster
An ECS Task Definition
One or more ECS Service The Task Definition is a template where you define your application (Docker image, ressources requests, networking mode, etc.). The Service is the component that will deploy our Fargate instance(s) based on our task definition(s) in the newly created Cluster.

At Camptocamp, we're doing IAC (infrastructure as code) using mostly Terraform. In order to simplify the deployment of all the resources necessary for the implementation of these components, I created two distinct Terraform modules: one to create an ECS Cluster and one to create Services within an existing cluster.
They have been designed to be flexible and reusable, and we will take a closer look at them to find out what they do and how they work.

⚙️ Module: ECS Cluster

Firstly, I created a module aiming to deploy:

an ECS cluster
an associated VPC
the necessary IAM roles
a Cloudwatch Log Group
network components (internet gateway, subnets, routes)

To use this module, we must provide some inputs variables:

A project name
A project environment (optional)
A list of public subnets
A list of private subnets
A list of availability zones

Link:

camptocamp / terraform-aws-ecs-cluster

Terraform module used to create a new AWS ECS cluster with VPC, IAM roles and networking components

terraform-aws-ecs-cluster

Terraform module used to create a new AWS ECS cluster with VPC, IAM roles and networking components

View on GitHub

⚙️ Module: ECS Service Fargate

Then, this second module aims to deploy a Fargate Service in an existing ECS Cluster (in this case deployed with the previous module).
It will also create everything necessary to be able to access our service. Here is the full list of resources that will be created:

An ECS Fargate Service with its needed Security Group
An ALB (Application Load Balancer) also with a Security Group
A Target Group
An HTTP and HTTPS Listeners
A DNS Zone and Record to the ALB
An ACM Certificate with validation

Once again, this module requires some variables to be used but this time the list is a little bit longer so here are just the most important ones:

An application name
An ECS Cluster ID
An ECS Task Definition ressource (to define what will be deployed on this instance)
A DNS Zone and Host
A VPC's ID and CIDR Blocks
An application port
Public and private subnets ids

Link:

camptocamp / terraform-aws-ecs-service-fargate

Terraform module used to create a new Fargate Service in an existing ECS cluster with networking components (ALB, Target Group, Listener)

terraform-aws-ecs-service-fargate

Terraform module used to create a new Fargate Service in an existing ECS cluster with networking components (ALB, Target Group, Listener)

View on GitHub

🎓 How to

So, our use case is to have a serverless Blackbox Exporter deployed on AWS ECS using a Fargate instance in the eu-west-1 region.
Furthermore, it must be accessible only by https with a valid ssl certificate and with basic authentication.

In order to achieve that, we must add a Nginx sidecar container which will handle basic auth and proxying of the traffic to the Blackbox for authenticated entities.

There is a simple architecture diagram of what we will achieve:

First, we will begin by creating the ECS Cluster using terraform-aws-ecs-cluster module, so, with all nested resources (VPC, subnets, etc.).

# versions.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

provider "aws" {
  region     = "eu-west-1"
}

# main.tf

module "ecs-cluster" {
  source = "git@github.com:camptocamp/terraform-aws-ecs-cluster.git"

  project_name        = "ecs-cluster-blackbox-exporters"
  project_environment = "prod"
  availability_zones  = ["eu-west-1a", "eu-west-1b"]
  public_subnets      = ["10.0.0.0/24", "10.0.10.0/24"]
  private_subnets     = ["10.0.20.0/24", "10.0.30.0/24"]
}

As you can see, a minimum of two availability zones is required in order to create VPC subnets. You also need to provide at least two public and two private cidr blocks.

Now that we have declared the module which will create a fresh ECS cluster with all networking stuff associated with it, we can create the Task Definition of our Blackbox application task that we will need after to define the ECS Service.
A Task Definition is a template where we define the containers that we will be executed on our ECS service (docker image to run, port mapping, environments values, log configuration, etc.), the resources required (CPU / Memory), the network mode of the task (with Fargate we must use awsvpc mode), and much more!

So, as we saw earlier, we will need two containers:

A Blackbox-Exporter container which will have port 9115 exposed but inaccessible from the outside of the cluster.
A Nginx container which will be exposed to the internet on port 80 with a basic authentication. It will forward authenticated users to the Blackbox container. We will use this docker image which allows an easy configuration of basic auth using env vars.

We will use the Cloudwatch Log Group created by the ecs-cluster module for the logs of these two containers.
Furthermore, we will also use IAM users created by the module for execution and task role ARNs of our Task Definition.

# main.tf

resource "aws_ecs_task_definition" "blackbox_fargate_task" {
  family   = "blackbox-exporter-task"

  container_definitions = <<DEFINITION
  [
    {
      "name": "ecs-service-blackbox-prod-container",
      "image": "prom/blackbox-exporter:latest",
      "essential": true,
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "${module.ecs-cluster.cloudwatch_log_group_id}",
          "awslogs-region": "eu-west-1",
          "awslogs-stream-prefix": "ecs-service-blackbox-exporter-prod"
        }
      },
      "portMappings": [
        {
          "containerPort": 9115
        }
      ],
      "cpu": 256,
      "memory": 512
    },
    {
      "name": "ecs-service-nginx-prod-container",
      "image": "beevelop/nginx-basic-auth:v2021.04.1",
      "essential": true,
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "${module.ecs-cluster.cloudwatch_log_group_id}",
          "awslogs-region": "eu-west-1",
          "awslogs-stream-prefix": "ecs-service-nginx-exporter-prod"
        }
      },
      "environment": [
        {
          "name": "HTPASSWD",
          "value": "${var.blackbox_htpasswd}"
        },
        {
          "name": "FORWARD_HOST",
          "value": "localhost"
        },
        {
          "name": "FORWARD_PORT",
          "value": "9115"
        }
      ],
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80,
          "protocol": "tcp"
        }
      ],
      "networkMode": "awsvpc"
    }
  ]
  DEFINITION

  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  memory                   = "512"
  cpu                      = "256"
  execution_role_arn       = module.ecs-cluster.ecs_task_execution_role_arn
  task_role_arn            = module.ecs-cluster.ecs_task_execution_role_arn

  tags = {
    Name        = "ecs-service-blackbox-exporter-td"
    Environment = "prod"
  }
}

In this exemple, I get the htpasswd from a Terraform variable var.blackbox_htpasswd. You can define it like this:

# variables.tf

variable "blackbox_htpasswd" {
  type = string
  sensitive = true
}

Next, we will need a DNS Zone where the ECS Service module will create the record.

# dns.tf

resource "aws_route53_zone" "alb_dns_zone" {
  name              = "example.com"
  delegation_set_id = "<Delegation_set_id>"
}

Optionally, you can create a delegation set in your AWS account, if you don't already have one, and add a delegation set id on your Route53 zone resource in order to always have the same DNS servers.

Finally, we can now create our ECS Service :

module "ecs-cluster-service-blackbox" {
  source = "git@github.com:camptocamp/terraform-aws-ecs-service-fargate.git"

  app_name        = "ecs-service-blackbox"
  app_environment = "prod"
  dns_zone        = "example.com"
  dns_host        = "blackbox.example.com"

  vpc_id          = module.ecs-cluster.vpc_id
  vpc_cidr_blocks = module.ecs-cluster.vpc_cidr_blocks

  ecs_cluster_id         = module.ecs-cluster.ecs_cluster_id
  task_definition        = aws_ecs_task_definition.blackbox_fargate_task
  task_lb_container_name = "ecs-service-nginx-prod-container"
  task_lb_container_port = 80

  subnet_private_ids = module.ecs-cluster.private_subnets.*.id
  subnet_public_ids  = module.ecs-cluster.public_subnets.*.id

  generate_public_ip = true

  depends_on = [
    aws_route53_zone.alb_dns_zone
  ]
}

As you can see, you must provide to the module some of the previously created resources including: vpc id and cidr_blocks, ECS cluster id, DNS zone, the task definition ressource and the subnets.
You must also set on which container the load balancer will redirect requests and on which port.

Once all your resources are properly configured, you can run a terraform apply to create them.

That's it 🥳! You now have a nice serverless Blackbox accessible on blackbox.example.com with basic auth! 🎉

Towards a Modular DevOps Stack

Raphaël Pinson — Wed, 23 Feb 2022 17:37:45 +0000

A year and a half ago, our infrastructure team at Camptocamp was faced with an increasingly problematic situation. We were provisioning more and more Kubernetes clusters, on different cloud providers. We used Terraform to deploy the infrastructure itself, and we had started to adopt Argo CD to deploy applications on top of the cluster.

We quickly ended up with many projects using similar logic, often borrowed from older projects, and most of these cluster were starting to use divergent code.

We thought it was time to put together a standard core in order to:

provision Kubernetes clusters;
deploy standard applications sets (monitoring, ingress controller, certificate management, etc.) on them;
provide an interface for developers to deploy their applications in a GitOps manner;
ensure all teams used similar approaches.

The DevOps Stack was born!

🌟 The Original Design

The original DevOps Stack design was monolithic, both for practical and technical reasons.

After all, we were trying to centralize best practices from many projects into a common core, so it made sense to put everything together behind a unified interface!

In addition to that, all the Kubernetes applications were created using an App of Apps pattern, because we had no ApplicationSets and no way to control Argo CD directly from Terraform.

As a result, the basic interface was very simple, for example:

module "cluster" {
  source = "git::https://github.com/camptocamp/devops-stack.git//modules/eks/aws?ref=v0.54.0"

  cluster_name = local.cluster_name
  vpc_id       = module.vpc.vpc_id

  worker_groups = [
    {
      instance_type        = "m5a.large"
      asg_desired_capacity = 2
      asg_max_size         = 3
    }
  ]

  base_domain     = "example.com"

  cognito_user_pool_id     = aws_cognito_user_pool.pool.id
  cognito_user_pool_domain = aws_cognito_user_pool_domain.pool_domain.domain
}

However, it could get very complex when application settings needed to be tuned!

With time, this architecture started being problematic for various reasons:

it was hard to deactivate or replace components (monitoring, ingress controller, etc.), making it hard to extend the core features;
default YAML values got quite mixed up and hard to understand;
as a result, overriding default values was unnecessarily complex;
adding new applications was done using extra_apps and extra_applicationsets parameters, which were monolithic and complex.

It was time to rethink the design.

🐙 Argo CD Provider

In order to escape the App of Apps pattern, we started looking into using a Terraform provider to control Argo CD resources. After various contributions, the provider was ready for us to start using in the DevOps Stack.

🗺 The Plan for Modularization

Using the Argo CD provider allowed us to split each component into a separate module. In a similar way to DevOps Stack v0, each of these modules would provide Terraform code to set up the component, optionally with:

cloud resources required to set up the component;
Helm charts to deploy the application using Argo CD.

💡 A Full Example

As a result, the user interface is much more verbose. The previous example would thus become:

module "cluster" {
  source = "git::https://github.com/camptocamp/devops-stack.git//modules/eks/aws?ref=v1.0.0"

  cluster_name = var.cluster_name
  base_domain = "demo.camptocamp.com"
  vpc_id       = module.vpc.vpc_id

  cluster_endpoint_public_access_cidrs = flatten([
    formatlist("%s/32", module.vpc.nat_public_ips),
    "0.0.0.0/0",
  ])

  worker_groups = [
    {
      instance_type        = "m5a.large"
      asg_desired_capacity = 2
      asg_max_size         = 3
      root_volume_type     = "gp2"
    },
  ]
}

provider "argocd" {
  server_addr = "127.0.0.1:8080"
  auth_token  = module.cluster.argocd_auth_token
  insecure = true
  plain_text = true
  port_forward = true
  port_forward_with_namespace = module.cluster.argocd_namespace

  kubernetes {
    host                   = module.cluster.kubernetes_host
    cluster_ca_certificate = module.cluster.kubernetes_cluster_ca_certificate
    token = module.cluster.kubernetes_token
  }
}

module "ingress" {
  source = "git::https://github.com/camptocamp/devops-stack-module-traefik.git//modules/eks"

  cluster_name     = var.cluster_name
  argocd_namespace = module.cluster.argocd_namespace
  base_domain      = module.cluster.base_domain
}

module "oidc" {
  source = "git::https://github.com/camptocamp/devops-stack-module-oidc-aws-cognito.git//modules"

  cluster_name     = var.cluster_name
  argocd_namespace = module.cluster.argocd_namespace
  base_domain    = module.cluster.base_domain

  cognito_user_pool_id     = aws_cognito_user_pool.pool.id
  cognito_user_pool_domain = aws_cognito_user_pool_domain.pool_domain.domain
}

module "monitoring" {
  source = "git::https://github.com/camptocamp/devops-stack-module-kube-prometheus-stack.git//modules"

  cluster_name     = var.cluster_name
  oidc             = module.oidc.oidc
  argocd_namespace = module.cluster.argocd_namespace
  base_domain    = module.cluster.base_domain
  cluster_issuer = "letsencrypt-prod"
  metrics_archives = {}

  depends_on = [ module.oidc ]
}

module "loki-stack" {
  source = "git::https://github.com/camptocamp/devops-stack-module-loki-stack.git//modules/eks"

  cluster_name     = var.cluster_name
  argocd_namespace = module.cluster.argocd_namespace
  base_domain      = module.cluster.base_domain

  cluster_oidc_issuer_url = module.cluster.cluster_oidc_issuer_url

  depends_on = [ module.monitoring ]
}

module "cert-manager" {
  source = "git::https://github.com/camptocamp/devops-stack-module-cert-manager.git//modules/eks"

  cluster_name     = var.cluster_name
  argocd_namespace = module.cluster.argocd_namespace
  base_domain      = module.cluster.base_domain

  cluster_oidc_issuer_url = module.cluster.cluster_oidc_issuer_url

  depends_on = [ module.monitoring ]
}

module "argocd" {
  source = "git::https://github.com/camptocamp/devops-stack-module-argocd.git//modules"

  cluster_name   = var.cluster_name
  oidc           = module.oidc.oidc
  argocd         = {
    namespace = module.cluster.argocd_namespace
    server_secrhttps://kubernetes.slack.com/archives/C01SQ1TMBSTetkey = module.cluster.argocd_server_secretkey
    accounts_pipeline_tokens = module.cluster.argocd_accounts_pipeline_tokens
    server_admin_password = module.cluster.argocd_server_admin_password
    domain = module.cluster.argocd_domain
  }
  base_domain    = module.cluster.base_domain
  cluster_issuer = "letsencrypt-prod"

  depends_on = [ module.cert-manager, module.monitoring ]
}

☸ The Cluster Module

As you can see, the main cluster module —which used to do all the work— is now only responsible for deploying the Kubernetes cluster and a basic Argo CD set up (in bootstrap mode).

🐙 The Argo CD Provider

We then set up the Argo CD provider using outputs from the cluster module:

provider "argocd" {
  server_addr = "127.0.0.1:8080"
  auth_token  = module.cluster.argocd_auth_token
  insecure = true
  plain_text = true
  port_forward = true
  port_forward_with_namespace = module.cluster.argocd_namespace

  kubernetes {
    host                   = module.cluster.kubernetes_host
    cluster_ca_certificate = module.cluster.kubernetes_cluster_ca_certificate
    token = module.cluster.kubernetes_token
  }
}

This provider is used in all (or most at least) other modules in order to deploy Argo CD applications, without going through a monolithic/centralized App of Apps.

🧩 Component Modules

Each module provides a single component, using Terraform and the Argo CD provider (optionally).

There are common interfaces passed as variables between modules. For example, the oidc variable can be provided as an output from various modules: Keycloak, AWS Cognito, etc. This oidc variable can then be passed to other component modules to configure the component's authentication layer.

In a similar fashion, the ingress module, which was so far only using Traefik, can now be replaced by another Ingress Controller implementation.

🐙 The Argo CD Module

The Argo CD module is a special one. A special entrypoint (called boostrap) is used in the cluster module to set up a basic Argo CD using Helm.

The user is then responsible for instantiating the Argo CD module a second time at the end of the stack, in order for Argo CD to manage itself and configure itself properly —including monitoring and authentication, which cannot be configured before the corresponding components are deployed.

🥾 The Bootstrap Pattern

Eventually, other modules might adopt the bootstrap pattern in order to solve chicken-and-egg dependencies.

For example, cert-manager requires Prometheus Operator CRDs in order to be monitored. However, the Kube Prometheus Stack might require valid certificates, thus depending on cert-manager being deployed. This could be solved by deploying a basic cert-manager instance (in a bootstrap endpoint), and finalizing the deploying at the end of the stack.

🚀 To Infinity

The modular DevOps Stack design is the current target for release 1.0.0.

While this refactoring is still in beta state, it can be tested by using the v1 branch of the project. You can also find examples in the tests directory (though not all distributions are ported yet).

Feedback is welcome, and you can contact us on the #camptocamp-devops-stack channel of the Kubernetes Slack.

Integrate an Application with Prometheus Operator and Package with a Helm Chart

Julien Acroute — Mon, 19 Jul 2021 08:59:28 +0000

In the previous posts, we saw:

how to implement metrics in applications
how to run the monitoring stack locally
how to test and debug metrics generated by a simple Python Flask application

In this post we will:

use Kubernetes Custom Resources to integrate our application with the Prometheus Operator
define some alerts based on the metrics generated by the application
deploy a custom dashboard in Grafana
package everything in a Helm chart, including a Grafana dashboard

Connect Prometheus to your Application

Prometheus will retrieve metrics from Pods with a /metrics HTTP endpoint. If the Prometheus Operator is deployed in your Kubernetes Cluster, the discovery of the Pods is done by deploying one of the following custom Kubernetes objects:

Using ServiceMonitor

When a ServiceMonitor object is deployed, Prometheus will create one target per address defined in the Endpoints object linked to the Service. This means every Pod is in a ready status used by the Service.

For example, if you have the following Service and Deployment in your cluster:



apiVersion: v1
kind: Service
metadata:
  name: webapp
  labels:
    component: backend
    instance: app
    name: containers-my-app
  namespace: my-app
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: webapp
  selector:
    component: backend
    instance: app
    name: containers-my-app



apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  labels:
    component: backend
    instance: app
    name: containers-my-app
  namespace: my-app
spec:
  selector:
    matchLabels:
      component: backend
      instance: app
      name: containers-my-app
  template:
    metadata:
      labels:
        component: backend
        instance: app
        name: containers-my-app
    spec:
      containers:
      - name: app
        image: ghcr.io/camptocamp/course_docker_backend:python
        ports:
        - containerPort: 8080
          name: webapp

As defined in the Deployment’s spec.template.metadata.labels field, Pods will have the following labels:

component: backend
instance: app
name: containers-my-app

The Service has a selector that matches labels of the Pod. Therefore the Service will load balance traffic to Pods deployed by the Deployment.

ServiceMonitor objects also use a selector to discover which Services need to be monitored. Prometheus will scrape metrics from every Pods behind selected Services.

For example, to retrieve metrics from our Pods, we can deploy the following ServiceMonitor object:



apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: webapp
  labels:
    component: backend
    instance: app
    name: containers-my-app
    release: prom
  namespace: my-app
spec:
  namespaceSelector:
    matchNames:
    - my-app
  selector:
    matchLabels:
      component: backend
      instance: app
      name: containers-my-app
  endpoints:
  - port: http

Prometheus Operator will search for Services:

in the my-app namespace,
with the following labels:
- component: backend
- instance: app
- name: containers-my-app
with a port named: http

It will then use the Service selector to find Pods. As a result, one target per Pod will be created in the Prometheus configuration.

Using PodMonitor

PodMonitor objects use a selector to find Pods directly. No Service needs to be deployed.

For our Pods, we can use the following PodMonitor:



apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: webapp
  labels:
    acomponent: backend
    instance: app
    name: containers-my-app
    release: prom
  namespace: my-app
spec:
  namespaceSelector:
    matchNames:
    - my-app
  selector:
    matchLabels:
      component: backend
      instance: app
      name: containers-my-app
  podMetricsEndpoints:
  - port: webapp

The Prometheus operator will search for Pods:

in the my-app namespace,
with the following labels:
- component: backend
- instance: app
- name: containers-my-app
with a port named: webapp

For each Pod, a new target will be added to the Prometheus configuration.

Configure Alerts

Why use Alerts?

Gathering and storing metrics is very useful to investigate when something goes wrong. But there are often some modifications of one or a combination of metrics before a service becomes completely unusable.

A common example of this is remaining free disk space. Fixing hard thresholds with arbitrary values on disk space is usually inefficient (you might actually end up with 95%, 100% and 101% thresholds). What needs to be monitored is actually the estimated time left until the disk is full, which can be obtained by running a time regression on the metric.

Some other examples:

For an online store, having no purchases during the afternoon is unusual, maybe there is an issue that blocks users in the payment process.
If the ratio of HTTP responses with code 500 suddenly increases, investigation is needed.

This is the purpose of alerts: when such events are detected, the system notifies the right person, allowing you to keep your eyes off dashboards.

After investigating and finding the root cause, you should always ask yourself if you can build an alert to detect such a case. There is also the possibility of increasing the observability if some metrics are missing.

How to define Alerts?

The Prometheus Operator allows the definition of alerts with a custom Kubernetes object: PrometheusRule

In this custom resource, you can define multiple alerts:



apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: webapp
  namespace: my-app
  labels:
    component: backend
    instance: app
    name: containers-my-app
    app: kube-prometheus-stack
    release: prom
spec:
  groups:
    - name: app
      rules:
        - alert: NoProductViewSince1h
          expr: (view_total - view_total offset 1h) < 1
          for: 5m
          labels:
            severity: Critical

In the example above, only one alert is defined.

Find below what needs to be defined for usual cases for each alert:

alert: the alert name
expr: a PromQL expression that triggers an alert if it returns something. This is why most of the time a threshold is used. With the offset function we compute the views from the past hour, if the result is above the threshold 1, then an alert is created and pushed to AlertManager.
Optional labels: a set of labels, usually used for alert severity
Optional for: delays triggering the alert. The PromQL expression must return some sample during the duration of the field for, before an alert is triggered.

Troubleshooting

There are many selectors involved in this process:

selector on ServiceMonitor to find Services
selector on Service to find Pods
selector on PodMonitor to find Pods

There are also selectors to discover ServiceMonitor, PodMonitor, and PrometheusRule objects. Those selectors are defined in the Prometheus object using the following fields:

Discovery of ServiceMonitor:
- spec.serviceMonitorSelector
- spec.serviceMonitorNamespaceSelector
Discovery of PodMonitor:
- spec.podMonitorSelector
- spec.podMonitorNamespaceSelector
Dicovery of PrometheusRule:
- spec.ruleSelector
- spec.ruleNamespaceSelector

Check Selectors

If the target is not discovered by Prometheus:

Check that your ServiceMonitor or PodMonitor is deployed in a Namespace that matches the namespace selector in the Prometheus object.
Check that labels on your ServiceMonitor or PodMonitor match the selector in the Prometheus object.
Check that the selector on your ServiceMonitor or PodMonitor matches labels defined in the Service or Pod.
Check that the Service or Pod are deployed in the Namespace selected by the namespace selector defined in the ServiceMonitor or PodMonitor.

In order to check a label selector, you can use the -l option of kubectl. For example, to check the following selector in a ServiceMonitor:



  selector:
    matchLabels:
      component: backend
      instance: app
      name: containers-my-app

run the following command:



kubectl -l component=backend,instance=app,name=containers-my-app get service

Check Port Name or Number

Check that the port number or name matches a port defined in a Service or a Pod.

ServiceMonitor references either an incoming port defined in the Service or a Pod port:

ServiceMonitor.spec.endpoints.port references the name of a Service port: Service.spec.ports.name
ServiceMonitor.spec.endpoints.targetPort references a Pod port: Pod.spec.containers.ports.containerPort or Pod.spec.containers.ports.name

PodMonitor references port defined on Pod:

PodMonitor.spec.podMetricsEndpoints.port reference the name of a Pod port: Pod.spec.containers.ports.name

Note that Prometheus will only use Pods with a Ready state.

Create Grafana Dashboards as ConfigMaps

Grafana includes an auto discovery mechanism for dashboards. Any ConfigMap with a label grafana_dashboard=1 is loaded into Grafana.

The following ConfigMap will create a minimal dashboard in Grafana. Note that this ConfigMap needs to be deployed in the same Namespace as Grafana.



kind: ConfigMap
apiVersion: v1
metadata:
  name: my-dashboard
  labels:
    grafana_dashboard: "1"
data:
  dashboard.json: |
    { "title": "Product Views",
      "time": { "from": "now-6h", "to": "now" },
      "editable": false,
      "panels": [ {
          "gridPos": { "h": 9, "w": 12, "x": 0, "y": 0 },
          "id": 2,
          "targets": [ {
              "exemplar": true,
              "expr": "rate(view_total[2m])",
              "interval": "",
              "legendFormat": "{{product}}",
              "refId": "A"
            } ],
          "title": "Product View",
          "type": "timeseries"
        }
      ]
    }

Package Monitoring Objects with Applications using Helm Charts

Including monitoring objects in Application Helm Charts is a good way to maintain the observability layer of an application. The monitoring components can be versioned with application packaging. Also the deployment of monitoring can follow the same workflow as the application.

I will not explain how to package an application, but I’ll demonstrate how to include the following elements:

Dashboard: ConfigMap with dashboard code
Alerts: PrometheusRule with alerts definition
Metrics endpoint: PodMonitor or ServiceMonitor

In the chart’s values.yaml, add a new section for monitoring:



monitoring:
  alerts: true
  dashboard: true

The monitoring.alerts values controls the deployment of the PrometheusRule object. The deployment of the ConfigMap for the dashboard is controlled by monitoring.dashboard.

For the PodMonitor or ServiceMonitor objects, we can check if the Prometheus Operator is installed using the .Capabilities.APIVersions.Has function:



{{- if .Capabilities.APIVersions.Has "servicemonitor.monitoring.coreos.com/v1" }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
...
{{- end }}

Additionally, for alerts and dashboards, we can check the "values" set on the Helm release:



{{- if .Capabilities.APIVersions.Has "prometheusrule.monitoring.coreos.com/v1" }}
{{- if .Values.monitoring.alerts }}
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
...
{{- end }}
{{- end }}

A common workflow to maintain a dashboard is to:

edit dashboards in the Grafana Web UI
copy the JSON model from Web UI
paste the JSON to a file in the Helm Chart
commit and push modifications

⚠ If the JSON representation of the dashboard is stored in the ConfigMap code, you will have to indent the content properly:



kind: ConfigMap
apiVersion: v1
metadata:
  name: my-dashboard
  labels:
    grafana_dashboard: "1"
data:
  dashboard.json: |
    { "title": "Product Views",
      "time": { "from": "now-6h", "to": "now" },
      "editable": false,
      "panels": [ {
          "gridPos": { "h": 9, "w": 12, "x": 0, "y": 0 },
          "id": 2,
          "targets": [ {
              "exemplar": true,
              "expr": "rate(view_total[2m])",
              "interval": "",
              "legendFormat": "{{product}}",
              "refId": "A"
            } ],
          "title": "Product View",
          "type": "timeseries"
        }
      ]
    }

It is generally easier to store the dashboard code in a dedicated file and then load the contents in the ConfigMap with some Helm templating functions:



{{- if .Capabilities.APIVersions.Has "prometheusrule.monitoring.coreos.com/v1" }}

{{- if .Values.monitoring.dashboard }}

apiVersion: v1

kind: ConfigMap

…

data:


  dashboard.json: |


    {{ .Files.Get "dashboard.json" | trim | nindent 4 }}

{{- end }}

{{- end }}

Conclusion

Deploying monitoring stacks —Prometheus, Grafana, AlertManager, ElasticSearch, Loki, …— provides much more observability in a project, but at the cost of consuming more resources. Developers not using these features is a waste of resources. The project also has poor observability because only system metrics and maybe http metrics are retrieved. Adding application specific metrics and even business metrics allows you to build beautiful dashboards with colors and graphs that are very fun to report during boring review meetings.

Enable OpenShift login on ArgoCD from GitOps Operator

Philippe Bürgisser — Thu, 20 May 2021 12:48:47 +0000

Since few weeks now, the operator Red Hat OpenShift GitOps became GA and embbed tools like Tekton and ArgoCD.

When the operator is deployed, it provisions a vanilla ArgoCD which miss the OpenShift integrated login. In this post, we are going to review the steps to enable it.

Deploy and fine tune the Red Hat OpenShift GitOps

Follow the official documentation on the installation of the operator
Once the operator is deployed, go to the menu Operators>Installed Operators and click on the freshly deployed Red Hat OpenShift GitOps
Using the dropdown Actions on top right of the page, choose Edit Subscription
On the YAML code, under the spec level, enable the DEX feature to enable external authentication and click Save

...
spec:
  config:
    env:
      - name: DISABLE_DEX
        value: 'false'
...

oc patch subscription openshift-gitops-operator -n openshift-operators --type=merge -p='{"spec":{"config":{"env":[{"name":"DISABLE_DEX","Value":"false"}]}}}'

Configure ArgoCD to allow OpenShift authentication

Change the project to openshift-gitops
Go to the menu Operators>Installed Operators and click on Red Hat OpenShift GitOps, select tab Argo CD
On the ArgoCD instance list, click on the three dots at the very left of the openshift-gitops and select Edit ArgoCD
On the YAML code, under the spec level, update the DEX and RBAC section to match the following

...
spec:
  dex:
    openShiftOAuth: true
  rbac:
    defaultPolicy: 'role:readonly'
    policy: |
      g, system:cluster-admins, role:admin
    scopes: '[groups]'
...

oc patch argocd openshift-gitops -n openshift-gitops --type=merge -p='{"spec":{"dex":{"openShiftOAuth":true},"rbac":{"defaultPolicy":"role:readonly","policy":"g, system:cluster-admins, role:admin","scopes":"[groups]"}}}'

Monitor the pods being restared to apply the configuration and test your login