DEV Community: Artur Bartosik

Exposing public applications on AWS EKS with Traefik

Artur Bartosik — Fri, 12 Jan 2024 07:19:13 +0000

In this short guide, I'll walk you through the process of exposing applications on AWS EKS using Traefik. Despite Traefik often being perceived as having complex configurations and unclear documentation, I aim to equip you with a straightforward setup that you can easily apply to your solution. Let's streamline the process and make exposing applications on the cloud a hassle-free experience.

Prerequisites

preconfigure EKS Cluster with AWS Load Balancer Controller and External DNS installed - you can leverage my repo that will help you set up EKS cluster with eksctl - GitHub repo
Route53 domain with ACM Certificate

As always I encourage you to install kubectx + kubens and configure shorter aliases to navigate Kubernetes easily.

Install & Configure Traefik

All resources you can find in my GitHub repo

First, check if the Ingress Class has been added with installation of AWS Load Balancer Controller from my EKS setup repo.

kubectl get ingressclas

-------------------------
NAME      CONTROLLER         
alb       ingress.k8s.aws/alb

We use this Ingress Controller to provision the main NLB pointing to our Traefik proxy.

Before we install the official Traefik helm chart let's take a few minutes to look at the helm values file helm/traefik.yaml.

#1
ports:
  web:
    expose: false
  websecure:
    port: 8443
    expose: true
    exposedPort: 443
    protocol: TCP
    tls:
      enabled: false
  traefik:
    port: 9000
    expose: true
    exposedPort: 9000
    protocol: TCP
    tls:
      enabled: false
  postgres:
    port: 5432
    expose: true
    exposedPort: 5432
    protocol: TCP
  rabbitmq:
    port: 5672
    expose: true
    exposedPort: 5672
    protocol: TCP

#2
ingressRoute:
  dashboard:
    enabled: false

service:
  #3
  annotations:
    external-dns.alpha.kubernetes.io/hostname: traefik.<YOUR_DOMAIN>, rabbitmq.<YOUR_DOMAIN>, postgres.<YOUR_DOMAIN> #4
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443, 9000"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <YOUR_CERTIFICATE_ARN> #5
    service.beta.kubernetes.io/load-balancer-source-ranges: 0.0.0.0/0
globalArguments:
  - "--api.insecure=false"

#6
logs:
  general:
    level: INFO
  access:
    enabled: false

At the top we define the Trafik port (#1). There are the network entry points into Traefik. We see dedicated ports defined for PostgreSQL and AMQP for RabbitMQ. The web port responsible for HTTP has been disabled and we have only left the HTTPS websecure open. Next, we disable default Ingress Route for Traefik dashboard (#2). Later we will create custom route for it. At the Traefik service section (#3) we define specific annotations for External DNS and AWS Load Balancer Controller. Here you need to know that by using these annotations you configure the AWS NLB and DNS records for Route53. Adjust external-dns.alpha.kubernetes.io/hostname with your root domain (#4) and service.beta.kubernetes.io/aws-load-balancer-ssl-cert with ACM certificate ARN (#5).

For more information please refer to the documentation:

The logging section (#6) is obvious. I advise setting log level to debug during the first setup as the info level is not too verbose.

So now, after adjustment in helm/traefik.yaml you are ready to install Traefik:

helm repo add traefik https://helm.traefik.io/traefik
helm install traefik traefik/traefik --create-namespace --namespace=network --values=helm/traefik.yaml

Right after, you should see NLB being created with the following list of listeners.

DNS records should also appear in the Route53 console.

If something doesn't happen, look for the answer in logs.

kubectl logs -f aws-load-balancer-controller- -n kube-system
kubectl logs -f external- -n kube-system
kubectl logs -f traefik- -n network

The next step will be adding Traefik dashboard route.

With the installation of Traefik, its CRD's are installed. Check our first router definitions in network/traefik-dashboard-route.yaml . First, we add a k8s secret (#1) that will hold the credentials for the Basic Auth, then we define the BA as Traefik Middleware (#2) and finally we create an IngressRoute which will be our routing path (#3). Remember to change your domain in the Host rule.

#1
apiVersion: v1
kind: Secret
metadata:
  name: dashboard-basic-auth-creds
  namespace: network
type: kubernetes.io/basic-auth
stringData:
  username: admin
  password: admin

---
#2
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: dashboard-basic-auth
  namespace: network
spec:
  basicAuth:
    secret: dashboard-basic-auth-creds

---
#3
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: dashboard
  namespace: network
spec:
  entryPoints:
    - traefik
  routes:
    - kind: Rule
      match: Host(`traefik.<YOUR_DOMAIN>`) #4
      middlewares:
        - name: dashboard-basic-auth
          namespace: network
      services:
        - kind: TraefikService
          name: api@internal

Apply the dashboard route.

kubectl apply -f network/traefik-dashboard-route.yaml

After a few seconds, when Traefik react to adding a new definition, the dashboard should be available in the browser under the below URL (The trailing slash is mandatory):

https://traefik.<YOUR_DOMAIN>:9000/dashboard/

Install sample apps (RabbitMQ & PostgreSQL)

Now let's add some more apps on different ports to better check Traefik's capabilities.

helm repo add bitnami https://charts.bitnami.com/bitnami

helm install rabbitmq bitnami/rabbitmq --create-namespace --namespace=messaging -f helm/rabbitmq.yaml
helm install postgres bitnami/postgresql --create-namespace --namespace=database -f helm/postgres.yaml

Add Traefik routes for:

Postgres port 5432
AMQP RabbitMQ port 5672
HTTPS RabbitMQ management panel

kubectl apply -f network/postgres-route.yaml
kubectl apply -f network/rabbitmq-route.yaml

List all Traefik routes

kubectl get ingressroute,ingressroutetcps --all-namespaces

-------------------------
NAMESPACE   NAME                                          
messaging   ingressroute.traefik.containo.us/rabbitmq-http
network     ingressroute.traefik.containo.us/dashboard     

NAMESPACE   NAME                                            
database    ingressroutetcp.traefik.containo.us/postgres     
messaging   ingressroutetcp.traefik.containo.us/rabbitmq-amqp

You should see two HTTP routes (Traefik dashboard, RabbitMQ panel) and two TCP routes (PostgreSQL and AQMP RabbitMQ). This state should be reflected in the Traefik panel. I encourage you to explore it because it is really rich in information and intuitive.

The final test will be to check if routing to test applications works correctly.

To check PostgreSQL try to connect with:

psql -h postgres.<YOUR_DOMAIN> -p 5432 -U root
# pass 'root' password

To check RabbitMQ AMQP connectivity execute my python script. Update URL in utils/rabbitmq_ampq_test.py before:

python3 utils/rabbitmq_ampq_test.py

RabbitMQ admin panel should be accessible on:

https://rabbitmq.<YOUR_DOMAIN>

Final Thoughts

As you can see, working with Traefik can be simple. In our lab we needed a single AWS NLB to expose several applications from different Kubernetes namespaces and on different ports. It’s a good benchmark of simplicity. This sort of approach has also limitations, but it’s a different topic. Be aware that other are other solutions in this Ingress segment such as Nginx Ingress Controller and HAProxy Ingress Controller. As always, I encourage you to experiment and form your own opinion. However, I hope I have helped you get through the basic setup more conveniently.

API Caching with ElastiCache Redis & AWS Lambda

Artur Bartosik — Wed, 08 Mar 2023 22:49:29 +0000

In this article, we will not only explore the benefits of using Redis as a caching solution for a REST API, but I’ll also provide a practical example of how to implement Redis caching in a serverless environment using AWS Lambda with TypeScript. I’ll demonstrate how to call an external API and cache response using ElastiCache Redis, resulting in faster response times and improved reliability. By following my example, you will learn how to integrate Redis caching into your own serverless applications.

Code example

If you want to directly jump to the code sample and play on a living organism, the repo is available on my Github right below:

https://github.com/luafanti/elasticache-redis-and-lambda

Deployment Overview

The infrastructure in demo is provisioned using the CloudFormation template. The stack created with the default parameters will provide the following resources:

VPC with single public & private subnet, and other base VPC components
NAT Gateway
ElastiCache Redis
TypeScript Lambda & HTTP API Gateway

**Please note that stack includes components (NAT & Redis) that will incur hourly costs even if you have AWS Free Tier.

You can also create a stack in MultiAZ mode. Then the stack will create:

VPC with two public and two private subnets
2 NAT Gateway, one per private subnet
ElastiCache Redis in Multi-AZ mode - one master and one replica instance
this same TypeScript Lambda

In Multi-AZ mode costs will be doubled due to two instances of NAT and Redis

Install is simple with

npm install 
sam build
sam deploy

To remove whole stack

sam delete

Important points to clarify

Lambda needs to be deployed inside VPC if want to connect managed Redis. ElastiCache is a service designed to be used internally in VPC.
NAT Gateway is required in this setup. Lambda running inside a VPC is never assigned a public IP address, so it can't connect to anything outside the VPC - in this case our external API. NAT Gateway resolves this problem.
If you want to connect to Redis cluster e.g. from local CLI, you have to setup a VPN connection or bastion host.
Top-level await isn’t supported in this Lambda sample. It’s because the actual setup uses CommonJS packaging. To enable this feature it needs to configure esbuild to output modules files with .mjs extension.

In-Depth

REST APIs have become an integral part of our systems. GraphQL and gRPC often in some aspects can be good replacements for old-fashioned REST, but personally, I still can't imagine forcefully avoiding this kind of API. Anyway, REST is ubiquitous and in our systems, we often integrate our services with that way. Sometimes it's an internal API, sometimes it's an external API. In the latter case, it’s more difficult because we have no control over the performance and availability of this API.

The long response time of external API automatically increases the overall time of request handling in our service. The unavailability of external API, even more, impacts our app and required special handling. The antidote to all this evil may be an additional layer of caching... and this is where Redis comes into the game.

Why Redis as a cache?

The main reason for Redis is its high performance. Redis is considered one of the faster key-value databases. There are several reasons behind that efficiency:

RAM-based data store. RAM access is several orders of magnitude faster than HDD or even SSD access. Not to mention access over API, which is burdened with the highest latency.

Efficient data structures. Redis offers various data structures such as List, Sets, Hashes, Bitmaps, etc. All types are implemented in C and to allocate memory use custom wrapper for malloc called zmalloc. This allows Redis to choose different alloc libraries depending on the needs.
Event-driven architecture. Redis uses a reactor design pattern to multiplex I/O to handle thousands of incoming requests at the same time using just a single thread.

The additional reason in the case of AWS is that the Redis database is available here as a managed service - ElastiCache Redis. It simplifies configuration and maintenance, and shifts some of the responsibility to the provider.

Step-by-Step caching flow

Flow is very simple… ProxyLambda first checks if response from External API exists in cache. As a cache key in this simple example, I just use the full request path also with query params. In complex API I can recommend a more advanced strategy for key generation. If response object exists in Redis cache, Lambda returns it directly. Otherwise, Lambda calls External API as before but additionally saves this response to the Redis cache. Thanks to this, a subsequent request to ProxyLambda for this same resource, will be returned from cache instead calling External API.

Basically the two diagrams below should explain it all

There is still the question about invalidating records in cache. Here the strategy must fit requirements. In this demo, objects in the cache are eternal. One of solution would be to hardcode expiration time on save (redis.setEx(cacheKey, 86400, apiResponse). A more elegant way would be to create dedicated invalidation Lambda, which will remove objects from the cache when receiving an event that in External API some resource has been removed or modified.

Pros and Cons of the Solution

🟢 Better performance. Responses from Redis cache can be much faster than from External API. Latency is also stable and doesn't depend on the current API load.

🟢 Improved reliability. Our faced API can respond even if External API is down.

🟢 Less load on the External API as less requests reaches it.

🟡 Additional work on management, and maintenance of ElastiCache.

🟡 Additional cost of ElastiCache cluster.

Cold starts with SnapStart for Java Frameworks (Spring Boot vs Quarkus vs Micronaut)

Artur Bartosik — Fri, 17 Feb 2023 11:11:58 +0000

At the last re:Invent 2022 AWS gave a lot of attention to the term Serverless. The main Keynote of AWS CTO Dr. Werner Vogels was very saturated with asynchronous approach and event-driven architecture. Also, new announcements were very closely related to these topics e.g. Step Functions & Event Bridge improvements.

Apart from all these trendy announcements, AWS also shows that it is trying to invest in technologies very cross-sectionally. It doesn’t cut itself off from Java technology, among others. This is evidenced by one of the largest Serverless announcements - SnapStart.

Due to the fact that I am associated with Java technology from the beginning of my professional career, I was very excited about this announcement. I was aware that so far JVM-based languages are not the main players for Lambda functions. Mainly because of its long cold start.
I haven't checked in a long time to see if anything has improved. So I decided that the release of SnapStart is a good time to evaluate cold starts for Java and its frameworks - Spring Boot, Quarkus, and Micronaut.

What is SnapStart?

Typically, AWS Lambda set up a new execution environment each time a function is first invoked or when the function is scaled up to handle increased traffic. As you probably know applications written in Java before accepting traffic need some time to initialize and start-up. This is the nature of JVM. SnapStart has been created to address this issue.

When SnapStart is enabled Lambda ahead of time creates a snapshot of initialized execution environment (memory and disk state) and persists it in the cache for low-latency access. This eliminates the need for the function to spend time on initialization (when the event came), as Lambda can quickly resume from the persisted snapshot instead.

I said "ahead of time" because Snapshot creation happens when you publish a function version and SnapStart works only for the published version of the Lambda function (can’t just use $LATEST).
Normally Init phase is the stage during Lambda performs multiple tasks like preparing the runtime container, downloading function code, initializing it, and so on… and then moving to the next phase. Init phase is Limited to 10 seconds. When SnapStart is activated, the Init phase happens earlier - yes, yes… when you publish a function version. In this case, 10-second timeout doesn't apply. Snapshot initialization can take up to 15 minutes.

Someone more curious might ask how the snapshot of the initiated function is possible to create? The answer is hidden behind a few magic terms.
First of all - CRaC (Coordinated Restore at Checkpoint) - open source project led by OpenJDK. It's focused on creating Java API responsible for saving and restoring the state of a JVM, including the currently running application - so-called checkpointing. CRaC based on next key project - CRIU (Checkpoint/Restore in Userspace) - that allows application running on Linux system to be paused and restarted at some point later in time, potentially on a different machine. The last key piece of this SnapStart puzzle is Firecracker and his microVMs. SnapStart uses micro Virtual Machine (microVM) snapshots to checkpoint and restore full applications. Interestingly, it turns out that Amazon engineers from Firecracker and Corretto (AWS JDK distribution) teams were involved in CRaC project at the early stage.

This means that AWS has long since taken the first steps to address the cold start problem for Java. It confirms my thesis that AWS invests in breakthrough technologies and knows that Java and JVM are still important in the IT market.

But unfortunately, not everything is so beautiful. SnapStart and the methods on which it is based introduced some challenges - Due to the fact that it operates on a memory dump.

Randomness - all results of java.util.Random operations can be the same so use java.security.SecureRandom instead because Amazon handles it in Corretto. But if one of your dependencies uses the first one, you still be in trouble.
Connections keeping - state of connections that your function establishes during the initialization phase isn't guaranteed when Lambda resumes from a snapshot. In most cases, network connections that an AWS SDK establishes automatically resume but for other connections you need to handle it on your own.
Stale credentials - the created snapshot also caches things like injected secrets and passwords (of course the whole snapshot is encrypted). Passwords can be rotated automatically. However, the snapshot can be used for a long period of time and not know anything about the password change. Our snapshot is immutable so will continue to use stale credentials. This applies not only to secrets. You need to protect yourself against such a case for any frequently changed data that you pull from an external sources into function memory. But don’t worry, you have tools to handle it e.g. post-snapshot hook.
Other lacks of support for AWS Lambda - provisioned concurrency, arm64 architecture, EFS, larger ephemeral storage (max 512 MB)

I'm curious if SnapStart will remain a functionality reserved only for Java runtime or maybe it will turn out that AWS prepares a similar trick for other runtimes. Presumably, other runtimes can't use snapshotting in quite the same way as JVM and others wouldn’t make sense to even attempt eg. Golang. But, I would like to see a SnapStart kind of solution for Node, which also can have hiccups in cold start ... especially with a large number of dependencies.

Spring Boot vs Quarkus vs Micronaut - introduction of competitors

Before we get to the merits, a brief introduction of competitors. Overall, all 3 frameworks are similar in terms of functionality and are suitable for building web apps and microservices, but they have different design goals and trade-offs.

BTW all sources and codes, as always, can be found on my GitHub

Spring Boot

Spring Boot is the most widely adopted and well-established of the three frameworks. Pivotal product has also the largest and most active community. It has been around for over a decade. It provides a wide range of features and is highly configurable, making it a good choice for large and complex applications. However, Spring Boot is more resource-intensive than Quarkus and Micronaut. Spring Boot uses a traditional, Just-in-Time (JIT) compilation approach, which can result in longer startup times compared to Quarkus and Micronaut, which use Ahead-of-Time (AOT) compilation. AOT compilation pre-compiles the code at build time, resulting in faster startup times and smaller memory footprint. Also, runtime dependency injection adds some overhead and complexity to spring-based workloads.

Quarkus

Quarkus is a relatively new framework that aims to provide the same functionality as Spring Boot, but with a smaller footprint and faster startup time. A project initiated by RedHat was created to be used for native compilation for GraalVM. It aims to be effective platform for serverless, cloud, and Kubernetes environments. Quarkus uses Ahead-of-Time (AOT) compilation to reduce startup time and memory usage. The community strongly appreciates the speed and convenience of development. More and more projects boast of migrating microservice workloads from Spring Boot to Quarkus. At the end I can add that the documentation is really good.

Micronaut

Micronaut like Quarkus is a relatively newer framework, but it has been gaining popularity in recent years. It has a very spring-inspired programming model. It also uses Reactor (instead of Vert.x that Quarkus use). So if you are coming from a Spring world, in Micronaut you will find many similar patterns, techniques e.g. Mono and Flux from Reactor core. At the same time Micronaut aims to avoid downsides of Spring. It minimizes using reflections and proxies and doesn't use runtime bytecode generation. The source I found says that performance is a tiny bit better with Quarkus, but it's just negligible value.

Measurements and charts

Let's move on to the main point - measurements. They were the main reason for writing this article. I wanted to measure how a cold start looks in 2023 for Java and its most popular frameworks. How much SnapStart makes things better. If SnapStart also affect warm start? How resource changes (Lambda memory) affect a cold & warm start performance? I hope the charts and tables below will help you answer these questions.

Vanilla Java

Non SnapStart		Cold Start (ms)				Warm Start (ms)
memory (MB)	error rate	p50	p90	p99	max	p50	p90	p99	max
128	0%	754.9	790.8	826.9	904	11.3	37	228.2	275.4
256	0%	566.6	599.9	666.9	676.1	1.9	15.4	107.1	328.2
512	0%	549.4	474.3	502.1	529.5	1.6	8.4	50.9	97.3
1024	0%	426.1	445.5	466.1	489.7	1.6	3.3	20.6	25.5
4096	0%	301.7	327.7	415.9	450.4	1.5	2.5	13.2	21.1

SnapStart		Cold Start (ms)				Warm Start (ms)
memory (MB)	error rate	p50	p90	p99	max	p50	p90	p99	max
128	0%	705.3	773.3	817.8	896.2	17.4	52.6	268	479.7
256	0%	401.6	447.9	473.2	536.6	7.7	20.3	120.2	214.3
512	0%	231.9	261.9	311.7	1174.6	1.7	9.7	53.5	135.6
1024	0%	203.9	231.6	367.3	399.1	1.6	3.7	24.8	53.4
4096	0%	241.5	353.4	484.1	501.4	1.5	2.6	14	25.7

Java Cold start median

Java Warm start median

Spring Boot

Non SnapStart		Cold Start (ms)				Warm Start (ms)
memory (MB)	error rate	p50	p90	p99	max	p50	p90	p99	max
128	100%	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
256	10,5%	5584.1	6867.7	7119.3	7157.4	28.3	1135.7	3582.5	3808.8
512	0%	3515.4	3647.8	3725.2	3762.8	12.1	20.2	52.9	180.6
1024	0%	3396.6	3512.3	3599.6	3599.6	3.9	9.2	18.5	94.7
4096	0%	2366.4	2525.2	3127.5	3191.1	3.4	5	10.6	33.4

SnapStart		Cold Start (ms)				Warm Start (ms)
memory (MB)	error rate	p50	p90	p99	max	p50	p90	p99	max
128	100%	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
256	60.3%	3920	5027.8	5149.8	5173.7	2399.7	3717.8	3931.8	4141.4
512	0%	515.3	554.2	598.7	611.4	5.6	18.6	37.1	54.3
1024	0%	347.3	381.1	451.6	1270	3.8	9.1	17.1	32.3
4096	0%	350.4	417.3	604.7	641	3.6	5.7	16.9	65.1

Spring Boot Cold start median

Spring Boot Warm start median

Quarkus

Non SnapStart		Cold Start (ms)				Warm Start (ms)
memory (MB)	error rate	p50	p90	p99	max	p50	p90	p99	max
128	100%	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
256	0%	3452.7	3543.6	3732.7	3757.3	50.5	73.1	213.8	317.2
512	0%	2738.2	2818.7	2890	2899.4	16.5	34	93.1	189.5
1024	0%	2305.7	2387.7	2512.6	4079.8	5.8	11.4	14.5	70.2
4096	0%	1676.2	1823	2028.8	2046.3	4.4	7.7	18.4	42.6

SnapStart		Cold Start (ms)				Warm Start (ms)
memory (MB)	error rate	p50	p90	p99	max	p50	p90	p99	max
128	100%	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
256	0%	1918.5	1977.8	2034.1	2063.4	48.8	63.5	168.2	264.1
512	0%	1059.1	1115.8	1144.8	1148.2	16	33.7	94.6	172.5
1024	0%	583.38	622	690.4	711.2	5.5	13.6	37.9	64.3
4096	0%	455.7	498.6	556	566.1	4.3	7.4	20.2	57.2

Quarkus Cold start median

Quarkus Warm start median

Micronaut

Non SnapStart		Cold Start (ms)				Warm Start (ms)
memory (MB)	error rate	p50	p90	p99	max	p50	p90	p99	max
128	100%	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
256	0%	3758.9	3912.2	4362.3	4262.7	29.1	46.8	174.1	315.7
512	0%	3391.1	3626	3916.1	3941.4	9.3	18.9	55	151.1
1024	0%	3146.2	3357.5	3680.2	3723.9	3.6	9.6	25.7	82
4096	0%	2517.6	2628.2	2738.1	2881.9	3.2	4.6	12.7	45.9

SnapStart		Cold Start (ms)				Warm Start (ms)
memory (MB)	error rate	p50	p90	p99	max	p50	p90	p99	max
128	100%	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
256	0%	1725.4	1814.6	2257.8	2289.8	29.1	44.2	175.2	231.8
512	0%	677	729.7	798.4	809.1	11.4	20.5	65.8	91
1024	0%	468.6	518.9	626.3	1373.1	3.7	8.6	25.5	97.7
4096	0%	388.8	439.2	562.1	594.3	3.1	4.9	16.7	67.1

Micronaut Cold start median

Micronaut Warm start median

Spring Boot vs Quarkus vs Micronaut vs Java - cold start charts

Spring Boot, Quarkus, Micronaut, Java - cold start without SnapStart median

Spring Boot, Quarkus, Micronaut, Java - cold start with SnapStart enabled median

General conclusions and observations

only pure Java Lambda can be run with 128 MB configuration and handle traffic. Frameworks fail with java.lang.OutOfMemoryError
Spring Boot framework fails part of requests also with 256 MB configuration. This spoils chart presentation for 256 MB
function package size is the largest for Spring Boot (13,7 MB) and the smallest for Micronaut (11,8 MB) apart from pure Java that size is around 1MB
the biggest difference in performance can be observed when upgrading memory from 256 MB to 512 MB. This applies to all frameworks, cold/warm start
enabling SnapStart brings the greatest benefit for Spring Boot - almost x10 shorter cold start in few configurations. For Quarkus it is average x4 short and for Micronaut +/- x6
for all frameworks with SnapStart enabled cold start comes close to pure Java cold start which is a fantastic result
looking at the median graph, you can see that Quarkus had the shortest cold starts without SnapStart. On the other hand, with SnapStart enabled it performs worst
SnapStart doesn't significantly affect warm start. It's hard to say if it has at all

Spring Boot logging with Loki, Promtail, and Grafana (Loki stack)

Artur Bartosik — Fri, 06 Jan 2023 11:08:08 +0000

In the previous article, I presented how to setup a monitoring stack using Prometheus Operator and integrate it with a sample Spring Boot app. This post will be analogous to the previous one but will be about another important topic - logs.

We will use Spring Boot application in demo. However, you will be able to configure any other app following this article. The only thing you need to ensure is to configure your app to produce logs in JSON format.

Configure Spring Boot to produce JSON logs

This is a GitHub link to my demo app. It’s simple Spring Boot web app used to debugging various stuff. There are many ways to configure JSON logging in Spring Boot. I decided to use Logback because it is easy to configure and one of the most widely used logging library in the Java Community. To enable JSON logging we need to add below dependencies.



implementation("ch.qos.logback.contrib:logback-json-classic:0.1.5")
implementation("ch.qos.logback.contrib:logback-jackson:0.1.5")
implementation("org.codehaus.janino:janino:3.1.9")

Janino dependency additionally adds support for conditional processing in configuration file logback.xml. Thanks to this, we can make our configuration parameterized. Use standard logs output when running the application locally, but enable JSON logging only when the application is running in the Kubernetes Pod by injecting the proper env variable.

Here you can find my Logback configuration file. This is all that you have to configure in Spring Boot app to enable JSON logging. When you run applications with env JSON_LOGS_ENABLED=true logs should be printed in JSON format. More information about Logback Layouts - component responsible for formatting log messages.

Grafana Loki stack

Loki Stack consists of 3 main components:

Loki - log aggregation system responsible for storing the logs and processing queries.
Promtail - lightweight agent responsible for gathering logs and pushing them to Loki. You can compare it to Fluentbit or Filebeat.
Grafana - visualization layer responsible for querying and displaying the logs on dashboards.

Grafana Labs in its Helm repository provides chart that can install Loki stack also with other complementary tools like Logstash or Prometheus.

Quick installation with helmfile

If you haven't used helmfile yet, I strongly encourage you to check out this tool. In my previous article about monitoring, I described why helmfile is worth using. Here I just leave for you Installation Gist.

Firstly, clone my repo with Lok stack helmfile, and check how little configuration is needed to install all stuff. This is because the Loki stack installation comes with reasonably safe defaults whenever possible, so we have only to overwrite some crucial values.
To install it we need to exec single command.



helmfile apply -i

After a short while, you should see a message that you have successfully installed three releases.



UPDATED RELEASES:
NAME                 CHART                             VERSION
loki-stack           grafana-labs/loki-stack             2.8.9
grafana-dashboards   local-charts/grafana-dashboards     1.0.0
demo                 luafanti/spring-debug-app           1.0.0

Explore logs in Grafana and understand Promtail scraping

Establish a tunnel to Grafana and check if preinstalled dashboards with logs can show data. If you want to know how my local Chart grafana-dashboardsadds dashboards to Grafana, I refer you to the paragraph in my previous article.



# establish port to Grafana Service
kubectl port-forward -n logging svc/loki-stack-grafana 3000:80

# get Grafana Credentials from Secrets
kubectl get secrets -n logging loki-stack-grafana --template='{{index .data "admin-password" | base64decode}}'
kubectl get secrets -n logging loki-stack-grafana --template='{{index .data "admin-user" | base64decode}}'

You should be able to see logs like below in custom dashboard.

Thanks to custom variables that use labels, we can create various filters for the dashboard. You can look up my configuration of variables and extend it with an analogy way for your own needs. At the top, I marked the filter with detected pods in selected namespace. In the lower part, you can see a preview of all labels that are associated with a single log line. Most labels are meta information that Promtail adds during scraping targets. This part of the Promtail configuration provides it. In this section, I also marked a few labels that not comes out-of-the box e.g. leavel , class , thread . We added these labels using the Promtail json stage. You need to know that Promtail processes scraped logs in a pipeline. A pipeline is comprised of a set of stages. json stage is a parsing stage that reads the log line as JSON and accepts JMESPath expressions to extract data.



- job_name: custom-config
  pipeline_stages:
    - docker: {}
    - json:
        expressions:
          timestamp: timestamp
          level: level
          thread: thread
          class: logger
          message: message
          context: context
    - labels:
        level:
        class:
        context:
        thread:
    - timestamp:
        format: RFC3339
        source: timestamp
    - output:
        source: message

One very important thing❗. As you can see in the config, above json stage I have docker stage added. This stage can read properly logs from Kubernetes with docker container runtime. If nodes from your cluster use different container runtime e.g. containerd (quite popular for managed Kubernetes clusters e.g. EKS, AKS) you have to replace this stage with cri stage.



- job_name: custom-config
  pipeline_stages:
    - cri: {}
    - json:
### rest of config

BTW, docker container runtime has already become deprecated. His successor became containerd. It was designed by Docker as well. Containerd is simplified compared to its predecessor - offers a minimum set of functionality for managing images and executing containers on a node.

Dealing with multi-line stack traces of Java

If you have ever worked with Java application logs you know that stack traces are painful. Not only because they are often misunderstood, but also because they are multi-line. When a multi-line event is written to a log output, Promtail and also many other scrapers, will take each row as its own entry and send the separately to Loki. I have a good news for you. The prepared configuration solves this problem thanks to Spring Boot JSON log output and appropriate Promtail configuration. You can verify it easily by following the steps.



# establish tunnel to Spring Boot Service
kubectl port-forward -n sandbox svc/demo-spring-debug-app 8080:8080

# call Spring Boot debug endpoint to produce exception log
curl http://localhost:8080/logs/exception

Back to Grafana and verify if stack traces are printed correctly.

Closing words

In this and the previous article, we covered two very important aspects of application observability - monitoring and logging. The only thing missing for the full trinity is tracing. Tracing is especially important in microservices or nanoservice (Serverless) architecture. In this area, Grafana Labs also has an interesting solution that is worth checking out - Tempo. In the next part of this series, I will try to introduce this tool in a similar way.

Spring Boot monitoring with Prometheus Operator

Artur Bartosik — Fri, 30 Dec 2022 10:19:29 +0000

In this article, we will install a Prometheus Operator that will automatically detect targets for monitoring. If you have used Prometheus before, either without the Operator or outside of Kubernetes, you will see how the Operator and its CRDs can make Prometheus flexible and how many things can happen magically leverages Kubernetes capabilities.

We will use Spring Boot application in demo. However, you will be able to configure any other app following this article. If your stack isn’t a Spring Boot just skip the first paragraph

Prepare Spring Boot to expose Prometheus metrics

My demo app (GitHub Link) uses Spring Boot version 3, or more precisely the latest release from 2022, i.e. 3.0.1. The core monitoring component in Spring Boot is Actuator. If you remember the migration of Spring Boot from version 1 to 2, you’ll probably remember that update brought a lot of breaking changes in Actuator. Fortunately, in the case of version 3, no such changes have been made, so you can apply the following configurations to Spring Boot version 2.x.x

To expose metrics consumable for Prometheus, you need to add two dependencies. The first one enables Actuator features, the second one is Prometheus exporter by Micrometer.



implementation("org.springframework.boot:spring-boot-starter-actuator")
runtimeOnly("io.micrometer:micrometer-registry-prometheus")

All you have to configure to enable default metrics is to provide the below configuration. As you can see, I expose entire Actuator with another port number. It is good practice to separate on the port level the business layer from the technical stuff.



management:
  server:
    port: 8081
  endpoints:
    web:
      exposure:
        include: "health,info,metrics,prometheus"

From now Spring Boot metrics in Prometheus format should be visible on http://localhost:8081/actuator/prometheus

Prometheus Operator

Kubernetes operators are applications that automate installation and configuration (Day-1 tasks) and scaling, upgrades, backups, recovery, etc. (Day-2 tasks) for stateful applications. We can say that Operators can replace part of manual administrator work. Under the hood, operators work in reconciliation loop (watch for changes in the application state) and use CRDs to extend the Kubernetes API. Generally speaking, it is the operational knowledge of a specific software contained in the custom controller code.

Prometheus Operator is an independent project from the Prometheus project. I know, it can lead to confusion. In the official README you can find short comparison. Basically, Prometheus Operator does what an operator should do - provides Kubernetes native deployment and management of Prometheus and related monitoring components like Grafana or Alert Manager.

Quick installation with helmfile

If you haven't used helmfile yet, I strongly encourage you to check out this tool. It provides a lot of improvements for working with Helm charts, but you don't need to go into all of them. You can easily switch and streamline your helm releases with helmfile and gain at the beginning one killer feature - interactive helm diff that works as Terraform plan. Installation Gist.

Firstly, clone my GitHub repo with Prometheus Operator helmfile, and check how little configuration is needed to install all stuff. This is because the Prometheus Operator installation comes with reasonably safe defaults whenever possible, so we have only to overwrite some crucial values.
To install it we need to exec single command.



helmfile apply -i

Flag -i apply interactive mode. Helmfile will ask for confirmation before attempting to modify cluster state. With the first installation, you will probably see a very loooooong diff status, so it won't be very useful. The power of this feature becomes apparent as you start adding small changes in your releases - the same as with Terraform.

After a short while, you should see a message that you have successfully installed three releases.



UPDATED RELEASES:
NAME                    CHART                                        VERSION
kube-prometheus-stack   prometheus-community/kube-prometheus-stack    43.2.0
grafana-dashboards      local-charts/grafana-dashboards                1.0.0
demo                    luafanti/spring-debug-app                      1.0.0

Establish tunnel to Grafana and check if preinstalled dashboards show some data.



kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

You should be able to see 3 dashboard directories.

First one - General is preinstalled together with Prometheus Operator, the remaining comes from local helm chart. This is the path where you can add any Grafana dashboard as a json file and install it along with whole stack. If you want to add your own dashboards, I recommend a method where you first import/create the chart in the Grafana UI, then export it as a json file, and then add it to the project. Thanks to this you will avoid the problem with missed datasource.

I could end this post here. We managed to install what we wanted so the goal was achieved 🎉 🎯. However, let me briefly explain the most interesting things that happen underneath.

How metrics flow from Spring Boot application to Grafana?

Thanks to Spring Boot Actuator project exposing operational information become trivial. As you can above, all metrics are exposed under a separate port 8081. Thanks to this, we have a dedicated gateway that we can open only for Prometheus. Actuator extended Prometheus exporter by Micrometer added dedicated endpoint where publish application metrics in Prometheus format /actuator/prometheus. We'll configure this endpoint to be polled by Prometheus (scraped) to fetch metrics and store them in its database. Note that in the Spring Boot app I added an additional label configuration. This adds application=spring-boot-demo label to every single metric. The label will be used in preinstalled Grafana dashboard as one of filter variable.

You need to know that Prometheus stores all data as time series. Every time series is uniquely identified by its metric name and optional labels. Labels enable dimensional data model. Any combination of labels for the same metric name identifies a particular dimension of that metric. Thanks to the fact that Prometheus stores metrics in this way, tools such as Grafana have an advanced ability to filter the results and present them also in various dimensions.

How Prometheus Operator discover endpoint to scrap?

This is a fundamental question that should be bothering us. If you've worked with Kubernetes before, you probably know that Prometheus requires configuring all endpoints for scraping. In Kubernetes environment, where pods appear and disappear quite often, it is impossible to provide such a static config. This is where one of powerful part of Prometheus Operator comes into play - ServiceMonitor. ServiceMonitor is one of Prometheus Operator CRDs. It defines a set of targets to be monitored by Prometheus. The Operator automatically generates scrape configuration based on that definition. Below you can see configuration responsible for defining ServiceMonitor for Spring Boot app.



additionalServiceMonitors:
    - name: kube-prometheus-stack-spring-boot
      selector:
        matchLabels:
          prometheus-monitoring: 'true'
      namespaceSelector:
        matchNames:
          - sandbox
      endpoints:
        - port: management
          interval: 5s
          path: /actuator/prometheus

It uses label selectors to define which Services to monitor, the namespaces to look for, and the port on which the metrics are exposed.



# check all installed ServiceMonitors
kubectl get servicemonitors.monitoring.coreos.com

Besides our explicitly defined ServiceMonitor, the default installation of the Prometheus Operator creates several others. These are ServiceMonitors used to monitor the Kubernetes cluster, and Prometheus or Grafana instances itself.

You can also view all targets defined by ServiceMonitor in the Prometheus UI.



kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
chrome http://localhost:9090/targets

One very important thing! Targets appear only when Prometheus finds a Service that has the appropriate labels - like here for my Spring Boot chart. If your Service doesn’t have the matched labels, you will not see the target marked as unavailable/down here, it simply will not appear here at all.

Where are Grafana dashboards installed?

When Grafana starts, it will update/insert all dashboards available in the configured path. Dashboards are provided under this path with help of sidecar container. Sidecar watch for new dashboards defined as a ConfigMap, then add a dashboard dynamically without restarting the pod. Below you can check the suitable configuration.



grafana:
  sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard
      folder: /tmp/dashboards
      provider:
        foldersFromFilesStructure: true

This is the first part of setup. We still need to somehow provide the definitions of our predefined dashboards as ConfigMap. For this I just created a local helm chart grafana-dashboards . As you can see, the only object that creates this chat is the ConfigMap with dashboard definitions which Grafana sidcar reads and extract under configured path.



# get all ConfigMaps with dashboard definitions
kubectl get cm | grep grafana-dashboards

# check if ConfigMaps are properly injected under configured path. You should see two dirs with predefined dashboards.
kubectl exec -it kube-prometheus-stack-grafana-5f4976649d-w7q56 -c grafana -- ls /tmp/dashboards

Switching between multiple versions of various tools

Artur Bartosik — Thu, 15 Dec 2022 11:48:55 +0000

I'm sure you've worked on different projects that used different versions of tools despite having the same technology stack. Whether you work as a frontend dev, backend dev, or DevOps, this problem can happen anywhere. Switching between versions of various tools can be tricky and painful. In this article, I decided to put together tools that will help reduce this pain.
I googled how to do it whenever I needed to switch between versions. This article I'll treat as a private cheat sheet. If you use the following tools on a daily basis, I encourage you to save this post and treat it as I do.

Switching Terraform version - tfenv

Installation

# Mac OS
brew install tfenv

Basic commands

# list all installed terraform versions
tfenv list

# list all available terraform versions for installation
tfenv list-remote

# install selected terraform version
tfenv install 1.3.6

# switch to installed terraform version
tfenv use 1.3.6

# print currently set terraform version
tfenv version-name

# uninstall selected terraform version
tfenv uninstall 1.0.11

Switching Java, Maven, and Gradle version - sdkman

Installation

curl -s https://get.sdkman.io | bash
source ~/.bash_profile

Basic commands

# list all installed and installable versions
sdk list java
sdk list maven
sdk list gradle

# install selected version
sdk install java 17.0.5-zulu
sdk install maven 3.8.6
sdk install gradle 7.6

# switch to installed version
sdk use java 17.0.5-zulu
sdk use maven 3.8.6
sdk use gradle 7.6

# print currently set version
sdk current java
sdk current maven
sdk current gradle

# uninstall selected version
sdk uninstall java 8.0.352-zulu
sdk uninstall maven 3.6.0
sdk uninstall gradle 7.4

# list all tools whose versions sdkman can manage
sdk list

Switching Node & npm version - nvm

Installation

# Mac OS
brew install nvm
mkdir ~/.nvm

# support for Oh My ZSH
echo "export NVM_DIR=~/.nvm\nsource \$(brew --prefix nvm)/nvm.sh" >> ~/.zshrc
source ~/.zshrc

Basic commands

# list all installed node versions
nvm ls

# list all available node versions for installation
nvm ls-remote

# install selected node version and switch to them
nvm install v19.2.0

# install latest LTS node version
nvm install --lts

# switch to installed node version
nvm use v19.2.0

# print currently set node version
nvm current

# uninstall selected node version
nvm uninstall v10.15.3

I will expand this article when I discover other tools this type. If you use any other version management tools, please let me know in the comment, I'd be happy to take a look.

Subjective comparison of Security Testing products. Sonatype vs JFrog vs Snyk

Artur Bartosik — Mon, 12 Dec 2022 15:21:17 +0000

For many years now, it has been impossible to imagine building solutions not relying on open source. In fact, every project I've worked on has benefited more or less from community development. This trend doesn’t apply only to product companies and start-ups. Large financial institutions and other critical sectors are also reaping the benefits of open source. The State of Open Source report by OpenLogic & Open Source Initiative, among others, confirms this statement.

The 2022 State of Open Source Report Open Source Usage, Market Trends, & Analysis by OpenLogic & Open Source Initiative

Such extensive use of open source can lead to some problems. Anyone who has worked with npm or maven based applications knows this. Developers, including myself, are often tempted to rely on external libraries and tools. This makes list of dependencies grow and grow, and it isn't easy to keep track of it. This raises the issue of trust in these dependencies. At some point, our project/product will need to generate reports with vulnerabilities and licenses of dependencies.

As software engineers, we should care about the quality and security of our solutions. For this reason, we should address this topic as early as possible in the Software development lifecycle (SDLC).

I have recently been researching and evaluating the most popular commercial products for open source security scanning - Sonatype, JFrog, Snyk. I have decided to bring all my outcomes together in this article. The comparison will be subjective and will refer to aspects that were crucial in my use case.

The technology stack of my project - a key aspect for further comparison:

Java/Kotlin microservices
Docker images as artifacts
GitHub as code repositories & CI/CD
Kubernetes
IaaC with Terraform

Under this Miro Link you will find the same table colored and better formatted than in limited Markdown.

	Sonatype	JFrog	Snyk
SaaS	No SaaS option. Planned for next years.	Available.	Available.
Self-hosted	Available.	Available.	Not available.
Free plan	Not available.	Available and sufficient for small projects but only for SaaS.	Sufficient for PoC and private home projects.
Pricing	Very high even for medium-sized teams due to minimum number of licenses required.	Reasonable for medium-sized teams. High cost with full package and more licenses.	Reasonable on any scale.
Licensing	Very complicated. A large number of tools and licenses create a confusing ecosystem. It's hard to find clear information in the documentation. Without contacting sales team, it is practically impossible to estimate the final price.Not possible to test anything without a trial license. To get it you have to arrange a series of meetings with sales (3 in my case) to starts with PoC - it costs a lot of time.	More complicated than Snyk due to several installations options (SaaS, Self-hosted) and a list of additional sub-products. Possibility to try a paid version after contacting the sales team.	The most transparent pricing. Possibility to test a paid plan for free without providing credit card and contact the sales team.
Docker images support	Not directly supported. In order to scan Docker image, it must be saved as tar archive. Link	Supported.	Supported.
CI/CD integrations	Not too good support. A lot of samples and instruction for old-fashioned Jenkins in offcial docs. For GitHub only not mantained community action. However, the list of supported CI/CD is growing, so I hope this will change in the future.	limited list of integrations. As for GitHub, you can use the official GitHub Action with the built-in JFrog CLI.	Support for most fo modern CI/CD’s including GitHub offcial Action and AWS Code Build.
IDE integrations	Provided for IntelliJ, VSC, and Eclipse with few shortcomings like worse dependency filtering for gradle projects.	Provided for VSC, most of JetBrains IDEs, and others niche players. For IntelliJ it has some nice features.	Provided for VSC, VS, JetBrains, and Eclipse. In IntelliJ very convenient and clear. Support also static code analysis.
CLI support	Available with few shortcomings. You have to always pass credentials to each command. Output returns a link to rich report in IQ Server but can’t generate well-formatted summary in CLI output.	Very powerful JFrog CLI. Great well-formatted summary of scanning in console output. Poor CLI documentation on the website.	Powerful CLI. Authentication with a Snyk account. CLI scan can export result in SARIF format!
Vulnerabilites information	Rich and well structured information about vulnerabilities. Usefull data about risk and the attack vector. Info how to determine if you are vulnerable. Recomendations how to fix or work around issue. Flagship copmonent called Version graph.	Has all important information about risk, attack vectors, and advices how to deal with vulnerability. Information is clear.	Probably the least informative of the other tools. However, there is everything that is most important. When it comes to the presentation layer, there aren’t too many bells and whistles here. It may be less readable due to small font used.
Open soruce licence scanning	Extensive possibility of managing licences. Black-listing of individual licences, manual verification, etc.	Just like Soantype, a wide range of license management options.	Very similar solution as in other tested tools. It seems to be poorer and less complicated, but therefore easier to use.
File reports	Possibility to generate PDF Report. Too bad you can't export to HTML. I can’t find option to generate ad-hoc report for more than one artifact/repo.	Possibility to generate PDF, JSON, or CSV report. No HTML option. You can include in report more than one project/artifact and apply advanced filters.	In Snyk web UI you can only export as CSV. CLI can produced JSON or SARIF scaning results.
Notifications & alerting	Limited number of built-in integrations for notification - only Email and JIRA. Fortunately, it is possible to configure Webhook.	Only Email & Webhook notifications.	Possible to setup Email alerting, built-in Slack notifications, custom Webhook or even JIRA integration.
Static code analysis	Not supported.	Not supported.	Supported with Snyk Code. Kotlin language is still in beta. With Snyk IaC you can also scan Kubernetes and Terraform configuration files.
Web browser UX	Mixed feelings. On the one hand, the new UI looks clear and not hurts eyes, on the other hand, it often took me a long time to find something. I think the main tabs could be better named and grouped. Somewhere you can still enable the old UI.	My first impression was good, but I tested JFrog first. Perhaps, if I had the opportunity to test it again I would have a more reliable opinion.	Well thought-out and intuitive. My only objection is to the clarity and smallness of the fonts.
Documentation	Poor impressions. In the documentation, you can find a lot of old, deprecated-looking content. Googling to find something, we often end up on a marketing page rather than proper documentation page.	As with Sonatype, the documentation looks poor. It's hard to find something navigating through their pages. Many articles, but poorly organized.	A great example of how documentation should look. Truly one of the best I've dealt with. Well structured. Navigating step by step from the most important things to the details. Lots of graphics and what developers like the most - code samples.

Summary

It is easy to see in the colored table that Snyk won my sympathy. IMHO, Snyk fits modern projects based on microservices, Docker, and CI/CD ala GitHub Actions. Sonatype turned out to be the least suitable for my project. However, I wouldn’t completely reject this tool. I think it has its advantages which you will see in projects that use Jenkins or private package repositories.

In the end, however, we didn't choose it. Instead, we decided to build our own security scanning pipeline based on open source solutions such as Trivy or Syft.

As I mentioned earlier, the above comparison is my personal assessment. My conclusions are very subjective as I did this research to find the right tool for my project. For a different tech stack, perhaps the conclusions could be different. I made the tool comparison in August 2022, so if you are reading this after a long time, know that some things may be out of date.

Injecting secrets from Vault into Helm charts with ArgoCD

Artur Bartosik — Wed, 07 Dec 2022 06:34:06 +0000

Managing secrets in Kubernetes isn’t a trivial topic. As is usual with Kubernetes, there are always many ways to achieve the desired goal and it’s often a problem to choose the right one for our case. In this article, I will show you one of ways to use ArgoCD with the help of Vault Plugin to inject secrets into Helm Charts. So, as you can guess, this may be one of the best ways to inject secrets if you are already using ArgoCD and Vault in your project.

Prerequisites

The basic knowledge of Kubernetes and Helm is obvious here. For ArgoCD and Vault I will try to guide you step by step in this article.
We will use the CLI wherever possible, so I recommend you install all the following:



# Helm
brew install helm

# Vault
brew tap hashicorp/tap
brew install hashicorp/tap/vault

#ArgoCD
brew install argocd

For structure, we will create 2 namespaces. The first one - technical, will be for Vault and ArgoCD instances, in the second one, we will install target Helm charts with injected secrets.



# namespace for Vault & ArgoCD
kubectl create ns toolbox

# namespace for resoruces installed by ArgoCD
kubectl create ns sandbox

I also encourage you to install kubectx + kubens to navigate Kubernetes easily.

Vault installation

For the beginning select toolboox namespace



kubens toolbox

To install Vault we will use the official Helm chart provided by HashiCorp. For simplicity, install it in developer mode. In dev mode, Vault doesn't need to be initialized or unsealed, but remember, it's only for development or experimentation. Never, ever run a dev mode in production



helm install vault hashicorp/vault --set "server.dev.enabled=true"

Vault can be configured via HTTP API, UI, or CLI. To operate Vault from local CLI establish port forwarding to vault-0 Pod, and setup Vault server address for already installed Vault CLI.



# port forwarding in separate terminal window
kubectl port-forward -n toolbox vault-0 8200

# login into Vault. Use 'root' token to authenticate into Vault
export VAULT_ADDR=http://127.0.0.1:8200
vault login

I also encourage you to explore browser UI interface of Vault. You have to use this same root token generated in dev mode.

Vault setup

Vault uses Secrets Engines to store, generate, or encrypt data. The basic Secret Engine for storing static secrets is Key-Value engine. Let’s create one sample secret that we’ll inject later into Helm Charts.



# enable kv-v2 engine in Vault
vault secrets enable kv-v2

# create kv-v2 secret with two keys
vault kv put kv-v2/demo user="secret_user" password="secret_password"

# create policy to enable reading above secret
vault policy write demo - <<EOF
path "kv-v2/data/demo" {
  capabilities = ["read"]
}
EOF

Now we need to create a role that will authenticate ArgoCD in Vault. We said that Vault has Secrets Engines component. Auth methods are another type of component in Vault but for assigning identity and a set of policies to user/app. As we are using Kubernetes platforms, we need to focus on Kubernetes Auth Method to configure Vault accesses. Let’s configure this auth method.



# enable Kubernetes Auth Method
vault auth enable kubernetes

# get Kubernetes host address
K8S_HOST="https://$( kubectl exec vault-0 -- env | grep KUBERNETES_PORT_443_TCP_ADDR| cut -f2 -d'='):443"

# get Service Account token from Vault Pod
SA_TOKEN=$(kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)

# get Service Account CA certificate from Vault Pod
SA_CERT=$(kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt)

# configure Kubernetes Auth Method
vault write auth/kubernetes/config \
    token_reviewer_jwt=$SA_TOKEN \
    kubernetes_host=$K8S_HOST \
    kubernetes_ca_cert=$SA_CERT

# create authenticate Role for ArgoCD
vault write auth/kubernetes/role/argocd \
  bound_service_account_names=argocd-repo-server \
  bound_service_account_namespaces=toolbox \
  policies=demo \
  ttl=48h

That's all for now in Vault. Once you have created all components, you can try to find them in the browser interface http://localhost:8200/

ArgoCD & Vault Plugin Installation

Time for the main actor of this article - Argo CD Vault Plugin It will be responsible for injecting secrets from the Vault into Helm Charts. In addition to Helm Charts, this plugin can handle secret injections into pure Kubernetes manifests or Kustomize templates. Here we will focus only on Helm Charts. Different sources required different installations, which you can find in plugin documentation.

What makes plugin documentation less clear is that it can be installed in two ways:

Installation via argocd-cm ConfigMap (old option, deprecated from version 2.6.0 of ArgoCD)
Installation via a sidecar container (new option, supported from version 2.4.0 of ArgoCD)

Since the old option will be not supported in future releases, I will install the ArgoCD Vault Plugin using a sidecar container. In order to properly install and configure ArgoCD, we need to follow a few steps:

Before all make sure you are still in toolbox namespace where we want to place Vault, ArgoCD, and all stuff for Vault plugin.



kubens toolbox

Create k8s Secret with authorization configuration that Vault plugin will use.



kind: Secret
apiVersion: v1
metadata:
  name: argocd-vault-plugin-credentials
type: Opaque
stringData:
  AVP_AUTH_TYPE: "k8s"
  AVP_K8S_ROLE: "argocd"
  AVP_TYPE: "vault"
  VAULT_ADDR: "http://vault.toolbox:8200"

Make sure you set the proper Vault address and role name.

Create k8s ConfigMap with Vault plugin configuration that will be mounted in the sidecar container, and overwrite default processing of Helm Charts on ArgoCD. Look carefully at this configuration file. Under init command you can see that we add Bitnami Helm repo and execute helm dependency build. It is required if Charts installed by you use dependencies charts. You can customize or get rid of it if your Charts haven’t any dependencies.



apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
data:
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
      init:
       command:
          - bash
          - "-c"
          - |
            helm repo add bitnami https://charts.bitnami.com/bitnami
            helm dependency build
      generate:
        command:
          - bash
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
            argocd-vault-plugin generate -s toolbox:argocd-vault-plugin-credentials -
      lockRepo: false

Finally, we have to install ArgoCD from the official Helm Chart but with extra configuration that provides modifications required to install Vault plugin via sidecar container.



repoServer:
  rbac:
    - verbs:
        - get
        - list
        - watch
      apiGroups:
        - ''
      resources:
        - secrets
        - configmaps
  initContainers:
    - name: download-tools
      image: registry.access.redhat.com/ubi8
      env:
        - name: AVP_VERSION
          value: 1.11.0
      command: [sh, -c]
      args:
        - >-
          curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
          chmod +x argocd-vault-plugin &&
          mv argocd-vault-plugin /custom-tools/
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

  extraContainers:
    - name: avp-helm
      command: [/var/run/argocd/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.4.8
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp-dir
        - mountPath: /home/argocd/cmp-server/config
          name: cmp-plugin
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin

  volumes:
    - configMap:
        name: cmp-plugin
      name: cmp-plugin
    - name: custom-tools
      emptyDir: {}
    - name: tmp-dir
      emptyDir: {}

# If you face issue with ArgoCD CRDs installation, then uncomment below section to disable it
#crds:
#  install: false

Save that Helm values as argocd-helm-values.yaml and execute below commands:



# once againe make sure to use proper namespace
kubens toolbox

# install ArgoCD with provided vaules
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n toolbox -f argocd-helm-values.yaml

All of the above configurations you can find in dedicated GitHub repo

If all went well, you should see similar list of Pods in toolbox namespace. Note that argocd-repo-server has sidecar container avp-helm

Install your resources with secrets injection

Is the time for a final check of our setup and installation of our Helm Charts.

Firstly, let’s try to authorize against ArgoCD. To obtain admin user password execute the below command:



kubectl -n toolbox get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

As with Vault, with ArgoCD we will also be working partly with the CLI and partly web UI.



# port forwarding in separate terminal window
kubectl port-forward svc/argocd-server 8080:80

# authorize ArgoCD CLI
argocd login localhost:8080 --username admin --password $(kubectl get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)

As our demo Chart we will use my debug Spring Boot application from GitHub repo. It’s simple web server that exposes a few debugging endpoints. Application has Helm templates and ArgoCD Application definition under /infra directory. To deploy this stack to k8s with Argo we need to apply ArgoCD Application CRD. Below full code sample, which you can also explore here.



apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo
spec:
  destination:
    namespace: sandbox
    server: https://kubernetes.default.svc
  project: default
  source:
    path: infra/helm
    repoURL: https://github.com/luafanti/spring-boot-debug-app
    targetRevision: main
    plugin:
      env:
        - name: HELM_VALUES
          value: |
            serviceAccount:
              create: true
            image:
              repository: luafanti/spring-boot-debug-app
              tag: main
              pullPolicy: IfNotPresent
            replicaCount: 1
            resources:
              memoryRequest: 256Mi
              memoryLimit: 512Mi
              cpuRequest: 500m
              cpuLimit: 1 
            probes:
              liveness:
                initialDelaySeconds: 15
                path: /actuator/health/liveness
                failureThreshold: 3
                successThreshold: 1
                timeoutSeconds: 3
                periodSeconds: 5
              readiness:
                initialDelaySeconds: 15
                path: /actuator/health/readiness
                failureThreshold: 3
                successThreshold: 1
                timeoutSeconds: 3
                periodSeconds: 5
            ports:
              http:
                name: http
                value: 8080
              management:
                name: management
                value: 8081
            envs:
              - name: VAULT_SECRET_USER
                value: <path:kv-v2/data/demo#user>
              - name: VAULT_SECRET_PASSWORD
                value: <path:kv-v2/data/demo#password>
            log:
              level:
                spring: "info"
                service: "info"
  syncPolicy: {}

You can see in line 54 & 56 placeholders with pattern <path:vault_secret_path#secret_key> where Vault Plugin will inject the actual value from Vault secret.
I also encourage you to compare this definition file with a definition without secret injection and without using Vault Plugin here. You should notice that source property is different when we use secrets injection. When we want to leverage on Vault plugin we need to define our Argo Application with source plugin and pass Helm Values using env HELM_VALUES

Let’s install this Argo Application and sync them.



# make sure you are in namespace where Argo has benn installed
kubens toolbox

# once you download soruce from GIT repo
kubectl apply -f infra/argocd/argocd-application-with-vault-secrets.yaml

# List ArgoCD applications
argocd app list

# Sync application
argocd app sync toolbox/demo

Once synchronization is finished, you should see beautiful green-full screen in ArgoCD UI

Verify if injection works.



# use port other than 8080 as the tunnel to Argo already uses this port
kubectl port-forward -n sandbox svc/demo-spring-debug-app 8090:8080

# check injected envs 'VAULT_SECRET_PASSWORD' 'VAULT_SECRET_USER' in debug app 
chrome http://localhost:8090/envs

One of the greatest things about the plugin is that if the value changes in Vault, ArgoCD will notice these changes and display OutOfSync status. Let's prove it.



# update secrets in Vault
vault kv put kv-v2/demo user="secret_user_new" password="secret_password_new"

# refresh application as well with target manifests cache
argocd app get toolbox/demo --hard-refresh

After Hard refresh you should see that your Argo Application back to status OutOfSync what is expected during Vault secret update. Thanks to this mechanism, you don't have to worry about losing control of keeping your secrets up to date.

Troubleshooting & possible problems

make sure your Vault secrets don’t disappear from Vault. In this guide, we use Vault in dev mode so secrets are stored in-memory. After cluster reboot all Vault objects will disappear.
if you would like to use different namespace names for Vault/ArgoCD etc. make sure you adjust your configuration files properly, especially HERE & HERE
Sometimes if you install ArgoCD multiple times in your cluster you can face error related to CRDs. You can uncomment this section to resolve it.

Vault Auto-unseal using Transit Secret Engine on Kubernetes

Artur Bartosik — Fri, 02 Dec 2022 08:35:56 +0000

Theoretical introduction

To make the Vault operational once it has been installed, we need to perform two actions:

Intialzie Vault
Unseal Vault

Unsealing has to happen every time Vault starts. This is because Vault starts with sealed state in which it can't read storage because it doesn't know how to decrypt it.

Initialzing happens once when the server started with new backend. During initialization Vault generates bunch of keys:

unseal keys
encryption keys
root token

As you can easily guess, unseal keys generated during initizlization are used for unsealing the Vault.

We can distinguish three options for unsealing the Vault:

Manual unsealing
Auto-unseal
Transit Unseal - de facto one of Auto-unseal option

Manual unsealing

Manual unsealing is the simplest and doesn't require any additional configuration. Vault generated root key (don’t be confused with root token) and uses an algorithm Shamir's Secret Sharing to split the key into chunks. During initialization, we can determine how many key shares will be needed to unseal the Vault. This is cloud agnostic and very flexible option but can become painful when you have many Vault clusters, many keys, and many key holders.

Auto-unseal

Auto-unseal reduces operational complexity and makes management less painful. In this approach, we delegate the responsibility of securing the unseal key from users to a trusted device or service. Vault with Auto-unseal, takes care of unsealing itself so we no longer need to worry about this as long as the service we have configured is available.

Services and devices supported with Auto-unseal you can find in official docs.

Transit Auto-unseal

We said that is one of Auto-unseal option. What makes this Auto-unseal different is that we don’t rely on an external service, but on external Vault itself. In this way, we can place one central Vault that will be responsible for Auto-unsealing other Vault instances.

Transit Auto-unseal setup

We are going to isolate our Vaults on the level of namespaces, so let's start with their creation.

# namespace for Vault central
kubectl create ns vault

# namespace for Vault with Transit Auto-unseal
kubectl create ns vault-a

Let’s start with installation of Vault Central. Save below Helm chart values to install Vault in HA mode with default manual unsealing.

server:
  affinity: ""
  ha:
    enabled: true
    replicas: 2
    raft: 
      enabled: true

# change namespace to Vault central
kns vault

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault -f vault-central-helm-values.yml

With the below initialization we split the root key into 4 shares (unseal keys). We can also set how many keys are required to reconstruct the root key, which is then used to decrypt the Vault's encryption key.

kubectl exec vault-0 -- vault operator init \
    -key-shares=4 \
    -key-threshold=2 \
    -format=json > vault-central-keys.json

Once is applied Vault generates unseal keys encrypted with base64 and hex and root token responsible for authentication against Vault. We will use it later.

{
  "unseal_keys_b64": [
    "4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T",
    "miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK",
    "EyVw9nQH/T+3zsa4HbPJ2s15l6B5MizMKQlKqs9taFzX",
    "zc7eU9MEvy9AaV4FPSQe7Jla2LcqSjS8KNPFDlQs0Rcg"
  ],
  "unseal_keys_hex": [
    "e169b9058b0d6a5fb331bb1bddec0d6e2eb32ed288397cf72fe3455fb8f0d3fdd3",
    "9a26ac837d4598f26ac7d2eb9cf69512e1badfd7ef8c0a99177829e1994ac3ec0a",
    "132570f67407fd3fb7cec6b81db3c9dacd7997a079322ccc29094aaacf6d685cd7",
    "cdcede53d304bf2f40695e053d241eec995ad8b72a4a34bc28d3c50e542cd11720"
  ],
  "unseal_shares": 4,
  "unseal_threshold": 2,
  "recovery_keys_b64": [],
  "recovery_keys_hex": [],
  "recovery_keys_shares": 0,
  "recovery_keys_threshold": 0,
  "root_token": "hvs.NbXRWfYNI4PmA860aBlC4onU"

Let’s unseal the first of two Vault instances in the central cluster. Pass two of four unseal_keys_b64 to the Vault to unseal them according to key-threshold.

kubectl exec vault-0 -- vault operator unseal 4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T
kubectl exec vault-0 -- vault operator unseal miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK

This same we have to do with the second instance, but before that, we must connect the second Vault to Raft storage cluster.

kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200

kubectl exec vault-1 -- vault operator unseal 4Wm5BYsNal+zMbsb3ewNbi6zLtKIOXz3L+NFX7jw0/3T
kubectl exec vault-1 -- vault operator unseal miasg31FmPJqx9LrnPaVEuG639fvjAqZF3gp4ZlKw+wK

Time to create Transit Secret Engine. This component will generate root key that we will use to Auto-unseal other Vaults.

# separate window
kubectl port-forward vault-0 -n vault 8200:8200

# set Vault address to use locally Vault CLI
export VAULT_ADDR=http://127.0.0.1:8200

# use 'root_token' generated during Vault initialization
vault login

# create transit secret 
vault secrets enable transit
vault write -f transit/keys/autounseal

Save below Vault policy that we will attach to Auto-unseal token.

path "transit/encrypt/autounseal" {
   capabilities = [ "update" ]
}

path "transit/decrypt/autounseal" {
   capabilities = [ "update" ]
}

# create policy with the above definition
vault policy write autounseal autounseal-policy.hcl

# create token for Auto-unsealing
$ vault token create -orphan -policy=autounseal -period=24h

Key                  Value
---                  -----
token                hvs.CAESIP_A7TaC9kt4yUeqg5_bJNiOJElb4UbA01xoV9Rk4ei6Gh4KHGh2cy5zVXpaa3A1MG9uOEZrNXN2a3J0TGl0cHU
token_accessor       wkTM4nsF0ehkRvIuBD9cedHC
token_duration       24h
token_renewable      true
token_policies       ["autounseal" "default"]
identity_policies    []
policies             ["autounseal" "default"]

Finally, we have created periodic orphan token which we will use for Auto-unsealing. Orphan means that created token doesn’t have a parent token so can’t be revoked together with ancestor. What is important to note, transit Auto-unseal token is renewed automatically by default.

Now it’s time to prepare Helm chart with the second Vault installation. It will be Vault with transit Auto-unsealing configuration. Check below Helm values file. Provide Vault central address and of course generated in previous step token (root key).

server:
  standalone:
    enabled: true
    config: |
      disable_mlock = true
      ui=true

      storage "file" {
        path = "/vault/data"
      }

      listener "tcp" {
        address     = "127.0.0.1:8200"
        tls_disable = "true"
      }

      seal "transit" {
        address = "http://vault.vault:8200"
        token = "hvs.CAESIP_A7TaC9kt4yUeqg5_bJNiOJElb4UbA01xoV9Rk4ei6Gh4KHGh2cy5zVXpaa3A1MG9uOEZrNXN2a3J0TGl0cHU"
        disable_renewal = "false"
        key_name = "autounseal"
        mount_path = "transit/"
        tls_skip_verify = "true"
      }

# change namespace to Vault Auto-unseal
kns vault-a

helm install vault hashicorp/vault -f vault-auto-unseal-helm-values.yml

The last step is to initialize Vault.

kubectl exec -it vault-0 -- vault operator init

Recovery Key 1: FFMLznSZq9wh/0CJwKLJWKkI9BrK/hjF6ySDYl9a19Ie
Recovery Key 2: qRfrdpkuEcXsF+dFh1Geru8VHkiL/hWUW+vY25twlwT1
Recovery Key 3: dX8sed7Dv8kI8kfFuYWDeQlagoikEVBpV5lZqH4ORnEh
Recovery Key 4: TCCplv+KvZHEOlICQU6eb67hGccufiqcZGkSiGlpQPkx
Recovery Key 5: ictL+c9czgMO+ME8qoTcGpgsvymEcORN7MkrpDE28x4a

Initial Root Token: hvs.6umGyyta9xrjq0q7Cv09Hr8X

Success! Vault is initialized

… and check its status to verify Sealed status.

kubectl exec -it vault-a -- vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.12.0
Build Date               2022-10-10T18:14:33Z
Storage Type             file
Cluster Name             vault-cluster-7a11a0ae
Cluster ID               a883d977-e70a-6367-3148-9c7a2c246897
HA Enabled               false

Each Vault initialization with configured to Auto-Unseal generates Recovery keys instead of Unseal Keys. Recovery keys can’t be used for unsealing Vault. These keys perform only authorization functions, which allows, for example, generates a new root token.

Final Thoughts

In the DevOps world, we want to automate everything possible and reduce operational complexity anywhere possible. Undoubtedly, Auto-unseal is something that fits into this assumption. However, from a security point of view, it is sometimes good to introduce a manual step with human intervention. In the above sample of Transit Auto-unseal we have exact combination of both approaches - Unsealed manually central cluster and related clusters Auto-unsealed by it.
It is also worth noting, that our solution is cloud-agnostic. We don’t rely on any external service, so we can setup it on-premise. The downside here is the introduction of a very crucial component in the overall deployment - central Vault cluster. Definitely, we have to think, how to ensure high availability and fault tolerance here.