DEV Community: Daniel German Rivera

Exploring Istio Ambient Mode: Understanding the Role of Istio-CNI

Daniel German Rivera — Wed, 29 Oct 2025 03:33:15 +0000

This article is part of my personal project, Smart-cash. Previous posts covered topics such as:

The Basics

Service Mesh: A dedicated infrastructure layer that manages service-to-service communication. It provides traffic routing, observability (metrics, logs, traces), security (mTLS), and resilience,all transparently.

Why Use a Service Mesh?

It simplifies microservices complexity by centralizing traffic control, enforcing security using mTLS, and enabling good observability. In my experience, it is good for environments with many interdependent services or strict traffic policies. For simple applications, it can add unnecessary overhead.

Istio: An open-source service mesh that integrates seamlessly with Kubernetes. It offers two deployment models:

The sidecar mode, adds an Envoy proxy to every pod.
The Ambient mode, eliminates sidecars and uses daemons running in nodes for traffic interception.

Istio Sidecar vs. Ambient Mode

In sidecar mode, Istio injects an Envoy proxy container into every pod that joins the mesh. You can configure this at the namespace or pod level. This proxy intercepts all inbound and outbound traffic. 3 main CRD can be highlighted:

VirtualService: Defines routing rules (path-based routing, retries, timeouts).
Gateway: Exposes services externally.
DestinationRule: Controls how to serve the traffic to the services (load balancing, circuit breaking).

In Ambient mode, Istio removes sidecars. Instead, it uses two daemons running in each node to intercept and secure traffic. This is an innovative way that uses kernel-level features.

Here the main components:

ztunnel Secures traffic and authenticates workloads within the mesh, manages mTLS, authentication, L4 authorization, and telemetry. This is only for L3 and L4 traffic; this doesn't terminate HTTP traffic. Ztunnel runs on every node.
IstioCNI Detects when a new pod is created and configures iptables rules to redirect traffic to ztunnel.
waypoint Provides L7 traffic management, HTTP routing, retries, etc. This is an optional component that is deployed per namespace or service.

Installing Istio

Installation will be done using Helm and Flux. For the detailed YAML manifest used, you can check these links:

Flux-Source Istio Helm repo
Flux-Helm-Release Istio Base (CRDs)
Flux-Helm-Release Istiod (Control Plane). Here, we need to set the profile to ambient, check the values
Flux-Helm-Release Istio-cni
Flux-Helm-Release Istio-ztunnel

You should see the pods in the istio-system namespace. Notice that only the control plane runs as a deployment; other components are daemon-sets.

Kiali is a useful tool for checking and visualizing the mesh. You can install using the documentation. For this case, the services graph looks like:

Add the following label to the namespace you want to include in the mesh. Here, we added it to the develop namespace:

istio.io/dataplane-mode=ambient

How does it work internally?

When a pod is created istio-cni automatically:

Detects the pod via Kubernetes CNI events
Enters the pod’s network namespace
Injects iptables rules to redirect inbound and outbound traffic to ztunnel

Let's check this in detail

WITHOUT Istio enabled

Inside one node,list the containers that are running:

crictl pods --namespace develop

We need to go inside the pod's network namespace and check the iptables rules. For that, we need to get the Pause container associated

sudo crictl inspectp 07689efcedf1f | jq -r '.info.pid'
sudo lsns -t net

With the PID of the Pause container, you can enter the network namespace or run a command directly:

sudo nsenter -t 18757 -n iptables -t nat -L | grep REDIRECT

The output should not show any redirection rules.

We can confirm that there are no redirection rules.

WITH Istio enabled

We can run the same commands to go inside pod network namespace, but when we list the IP tables rules, we will see these redirection rules:

Those ports are related to Istio Ztunnel, which are listed in Documentation.

With this, all traffic inside the pod is redirected to a ztunnel proxy, which manages it. This eliminates the need for a sidecar container.

IstioCNI plays a key role in Ambient Mode. It replaces the sidecar injection step by configuring traffic redirection at the network namespace.

In the next post, we will check mTLS and custom configurations.

Configuring Tracing with OpenTelemetry on AWS EKS and Go application

Daniel German Rivera — Fri, 07 Mar 2025 01:57:01 +0000

This article is part of a personal project called Smart-cash. Previous posts covered topics such as The deployment of AWS and Kubernetes resources and Configuring logging with FluentBit, among others.

Project Source Code

The full project code can be found here, the project is still under development, but you can find the terraform code to create AWS resources and also the Kubernetes manifests.

Key concepts

Two key concepts that often come up in modern system monitoring are observability and telemetry.

Observability helps us understand what is happening in the system.
Telemetry refers to the data generated by applications, which includes logs, metrics, and traces.

This post focuses on traces

What is distributed tracing?

Distributed tracing provides visibility into the path that requests follow through an application, helping to identify which parts of the system are experiencing errors and how much time each operation takes.

Imagine an application that generates two random numbers and stores them in a database, two complex functions handle the numbers calculation. To gain visibility into the system’s behavior, we can introduce tracing.

Here we can introduce another important concept: The Span.

A span represents a single operation within a trace, in the example, 3 spans can be defined:

One for each function that calculates a random number.
One for the database call.

By structuring the trace with these spans, we can better understand what happens at each step, identify bottlenecks, and debug issues more effectively.

Introduction to OpenTelemetry

OpenTelemetry(OTel) is an open-source, vendor agnostic tool for generating and managing telemetry data, such as traces, metrics, and logs. However, storage and visualization of this data must be handled with other tools.

OTel helps us instrument our applications by providing APIs to define how telemetry data is generated, as well as components that can receive and export this data to external endpoints.

OpenTelemetry collector

OpenTelemetry collector is a key component that works as a proxy to receive, process, and export telemetry data.

Detailed information about receivers, processors, and exporters can be found in OpenTelemetry Documentation.

The collector is not a mandatory component; data can be exported directly to the backend using libraries. However, doing so adds processing overhead to the application.

In this scenario, a collector will be installed, and the data will be sent to Jaeger.

Installing OpenTelemetry on an AWS EKS cluster

We will use OpenTelemetry Helm chart, installation is managed by FluxCD. All the files can be found in the repo.

Helm chart values

Let's start with some general values for the Helm chart

 mode: "deployment"
   namespaceOverride: "observability"
   presets:
     kubernetesAttributes:
       enabled: true
       extractAllPodLabels: true
       extractAllPodAnnotations: true 
   image:
     repository: otel/opentelemetry-collector-k8s
     pullPolicy: IfNotPresent
   command:
     name: "otelcol-k8s"

Let's focus in the presets section, which allows predefined configurations for specific scenarios. In this case, the kubernetesAttributes preset:

✔ Extracts all pod labels and annotations to enrich traces with Kubernetes metadata.
✔ Uses the Kubernetes Attributes processor to automatically add pod-related information to telemetry data.

This additional metadata helps correlate telemetry data with the Kubernetes environment.

Collector configurations

OpenTelemetry collector configuration is passed in the chart values, the configuration defines:

Receivers ---> Where telemetry data is received.
Processors ---> How the data is modified or filtered.
Exporters ---> Where the data is sent.

Let's break down the configuration

Receivers

The collector listens for telemetry data on port 4318 (HTTP). Applications should send telemetry data to this endpoint.

receivers:
  otlp:
   protocols:
     http:
      endpoint: $${env:MY_POD_IP}:4318

Processors

The processor manages data in batches using a default configuration ({}). This groups data before exporting it, helping to reduce network load.

processors:
  batch: {}
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80

The memory_limiter processor prevents high memory consumption by monitoring usage at intervals defined by check_interval and limiting usage based on limit_percentage.

In this case, the collector checks memory usage every 5 seconds.
If usage exceeds 80%, the collector drops data to prevent a crash.

Exporters

The collector will send the data to a K8 service using the internal DNS jaeger-traces-collector.observability.svc.cluster.local.

config:
  exporters:
    otlphttp:
     endpoint: "http://jaeger-traces-collector.observability.svc.cluster.local:4318"
     tls:
      insecure: true

Services section

service:
  pipelines:
   traces:
    receivers:
      - otlp
    exporters: 
      - otlphttp

The service section defines the enabled components(receiver, exporters, processors) and specifies how the data(traces, metrics, or logs) flows through pipelines

For this scenario:

Only traces are processed in the pipeline.
The exporters and receivers defined above are used.

Instrumenting a Golang microservice

At a high level, the code is structured as shown in the following diagram:

The Gin Web Framework is used to create the API. Incoming requests pass through different layers (handler, service, and repository), each responsible for specific logic.

OTel provides APIs and SDKs to generate and collect telemetry data. Additionally, there are several instrumentation libraries that simplify these tasks.

Gin instrumentation library will be used.

Init the OTel SDK

To begin, we need to create an OTel resource, which represents an entity producing telemetry data—in this case, the microservice.

A resource can have attributes that are configured during its creation. These attributes help in discovering telemetry data, particularly the traces generated by the application.

Let's see part of the code used here

res, err := resource.New(
   context.Background(),
   resource.WithFromEnv(),   // Discover and provide attributes from OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME environment variables.
   resource.WithTelemetrySDK(), // Discover and provide information about the OpenTelemetry SDK used.
   resource.WithContainer(),    // Discover and provide container information.
 resource.WithAttributes(semconv.ServiceNameKey.String("ExpenseService")), // Add custom resource attributes.
)

Next, we need to create an exporter to send the data to the previously created collector:

exporter, err := otlptracehttp.New(
    context.Background(),
    otlptracehttp.WithEndpoint(otelUrl+":4318"),
    otlptracehttp.WithInsecure(),
)

To start generating traces, we need to define a Tracer Provider responsible for generating and managing traces. Here we set some configurations.

tp := trace.NewTracerProvider(
        trace.WithBatcher(
            exporter,
            trace.WithMaxExportBatchSize(trace.DefaultMaxExportBatchSize),
            trace.WithBatchTimeout(trace.DefaultScheduleDelay*time.Millisecond),
            trace.WithMaxExportBatchSize(trace.DefaultMaxExportBatchSize),
        ),
        trace.WithResource(res),
    )

The complete code for initializing the collector can be found here

Configuring Gin middleware

router := gin.New()

router.Use(
    otelgin.Middleware("ExpenseService", otelgin.WithFilter(filterTraces)),
    gin.Recovery(), gin.Recovery(),
)

The key point in this code is the service_name(ExpenseService) passed to the middleware. It must remain consistent across all spans generated to ensure accurate and unified trace data.

Gin library will manage internally the instrumentation for this part of the code.

Creating spans

A trace is composed of multiple spans that can be nested using context.

Here’s how to create a span in one of our functions (service layer) used to create an expense:

func createExpense() {

tr := otel.Tracer("ExpenseService") # Trace Creation Microservice Name 

trContext, childSpan := tr.Start(ctx,"CreateExpense")
 childSpan.SetAttributes(attribute.String("component","serviceLevel"))

defer childSpan.End()

response, err := s.expensesRepository.CreateExpense(trContext, expense)
}

In the first line, a tracer is created with the name ExpenseService.

Then, a new span named CreateExpense is created and associated with this tracer. This span includes an attribute called component with the value serviceLevel.

The span creation returns:

trContext: A new context that carries span information for passing to other functions.
childSpan: The span object itself.

This allows you to track the execution and timing of different parts of the function.

Now, let’s proceed to the Jaeger installation to visualize the traces created.

Visualize traces with jaeger

Jaeger is an end-to-end distributed tracing system designed for monitoring and troubleshooting microservices-based architectures.

Commonly integrated as an OpenTelemetry backend, Jaeger stores and visualizes trace data, helping developers understand the flow and performance of their applications.

The Jaeger operator simplifies managing Jaeger resources on Kubernetes, the creation of the Jaeger instance is made by this yaml file:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-traces
  namespace: observability
spec:
  strategy: production
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx # Ingress annotations here
    ingressClassName: nginx
    hosts:
      - jaeger.smartcash.rootkit.site #your domain name.
  collector:
    maxReplicas: 5
    resources:
      limits:
        cpu: 100m
        memory: 128Mi

Through K8 ingress we can access the Jaeger UI and visualize the traces.

In the Jaeger UI, traces can be filtered based on the service name defined during OpenTelemetry (OTel) setup — in this case, expenses.

Jaeger displays the trace along with its spans, which represent individual operations within the trace. For this example, we have three spans:

Handler Span: Captures the execution of the HTTP handler.
Service Span: Covers the business logic executed in the service layer.
Repository Span: Represents the database connection and related operations. This structure helps in understanding the flow of requests and identifying potential bottlenecks or failures in the microservice.

Each span includes the component tag added in the code, along with metadata about the Kubernetes environment (such as pod labels and annotations). This enriched information helps in tracing requests across distributed components effectively.

Visualizing span created by Gin Library

The following image shows the span generated by the Gin Instrumentation library, all the data related to the incoming request is aggregated by this library.

Configuring Logging in AWS EKS Using Fluent Bit and CloudWatch

Daniel German Rivera — Sun, 06 Oct 2024 15:27:51 +0000

This article is part of a personal project called Smart-cash. Previous posts covered the deployment of AWS and Kubernetes resources and how to install FluxCD to implement GitOps practices.

Observability is essential for any application, and the Smart-cash project is no exception. Previously Prometheus was integrated for monitoring.

Project Source Code

The full project code can be found here, the project is still under development but you can find the terraform code to create AWS resources and also the Kubernetes manifest.

In this link you can find the files used in this post

Option 1. Export logs directly to Cloudwatch Logs(No Cloudwatch add-on)

The simplest configuration involves using Fluent-Bit's Tail Input, which reads the logs in the host /var/log/containers/*.log and sends them to Cloudwatch. This approach could be enough if you want to centralize the logs in CloudWatch or maybe another platform.

Fluent-Bit installation

The Fluent-Bit Helm chart will be used in combination with FluxCD.

Adding the FluxCD source:

kind: HelmRepository
metadata:
  name: fluent-bit
  namespace: flux-system
spec:
  interval: 10m0s
  url: https://fluent.github.io/helm-charts

Adding the Helm chart

The supported values for the Fluent-Bit Helm chart can be found here.

To use Fluent Bit with AWS, the following requirements must be met:

IAM Roles for Service Accounts (IRSA): You must set up an IAM role with permissions to create CloudWatch Logs streams and write logs. This role should be associated with the service account that Fluent Bit uses. AWS EKS Pod identity is also an option.
CloudWatch Log Group: You can either create the CloudWatch Log group in advance or allow Fluent Bit to handle the log group creation.

The main configuration is shown below (some lines have been omitted for brevity). The full configuration file can be found link

Fluent-Bit will run as a DaemonSet. The Helm chart will create the RBAC and the Service account, named fluent-bit, which will be annotated with the AWS IAM role with the appropriate CloudWatch permissions.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
###ommited-lines
spec:
  chart:
    spec:
      chart: fluent-bit
      version: 0.47.9
      ###ommited-lines
  values:
    kind: DaemonSet
    ###ommited-lines
    serviceAccount:
      create: true
      annotations: 
        eks.amazonaws.com/role-arn: arn:aws:iam::${ACCOUNT_NUMBER}:role/role-fluent-bit-${ENVIRONMENT}
    rbac:
      create: true
    ###ommited-lines

A volume is needed for Filesystem buffering, which helps to manage backpressure and overall memory control, also this is used to store the position (or offsets) of the log files being read, allowing Fluent Bit to track its progress and resume from the correct position if needed.

extraVolumes:
      - name: fluentbit-status
        hostPath:
          path: /var/fluent-bit/state
    extraVolumeMounts:
      - name: fluentbit-status
        mountPath: /var/fluent-bit/state

Configs section defines the Inputs, Filters, and Outputs for collecting and processing data. For this scenario an Input of Tail type is configured to read the content of the files located at /var/log/containers/*.log. Let's break down the configuration details:

Note: AWS has some advanced configurations for inputs, filters and output, you can check this link

The service section defines the global properties for Fluent-Bit, for this case:

Flush Interval: Set to 1 second, meaning Fluent Bit will send the collected logs to the configured output destinations every second.
Log Level: Set to Info, which includes informational messages as well as warnings and errors.
The storage path for Filesystem buffering is the volume mounted in previous configurations with a backlog memory limit of 5M, which means that if Fluent-bit service reaches this limit, it stops loading any more backlog chunks from the storage path into memory.

config:
  service: |
    [SERVICE]
      Daemon Off
      Flush  1
      Log_Level  info
      Parsers_File /fluent-bit/etc/parsers.conf
      Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
      HTTP_Server On
      HTTP_Listen 0.0.0.0
      HTTP_Port  \{\{ .Values.metricsPort \}\}
      Health_Check On
      storage.path  /var/fluent-bit/state/flb-storage/
      storage.sync              normal
      storage.checksum          off
      storage.backlog.mem_limit 5M

Inputs define the data sources that Fluent Bit will collect logs from. In this scenario, the Tail input is used, which allows Fluent Bit to monitor one or more text files. Key points for this configuration include:

In common Kubernetes environments, container runtimes store logs in the /var/log/pod/ and /var/log/containers/ (containers directory has symlinks to pod directory). Each log file follows a naming convention that includes key information like the pod name, namespace, container name, and container ID, for this case entries for fluent-bit, cloudwath-agent,kube-proxy, and aws-node will be ignored
Some log entries may span multiple lines. Fluent Bit handles multi-line logs with built-in modes, and for this scenario, the Docker or CRI modes are used to process them correctly.
To track the last line read from each log file, Fluent Bit uses a database to store this position. The database is saved in the previously mounted volume, ensuring that Fluent Bit can resume reading from the correct location.

[INPUT]
   Name                tail
   Tag                 applications.*
   Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
   Path                /var/log/containers/*.log
   multiline.parser    docker, cri
   DB                  /var/fluent-bit/state/flb_container.db
   Mem_Buf_Limit       50MB
   Skip_Long_Lines     On
   Refresh_Interval    10
   storage.type        filesystem
   Rotate_Wait         30

Outputs define where the collected data is sent, and Fluent-Bit provides a plugin to send logs to CloudWatch.

If you check the Input configurations there is a tag defined, applications.*. this helps to assign a label to the logs collected for that Input, in this case, it ensures that logs with this tag are routed to the specified output destination.
CloudWatch log groups can be created by Fluent Bit, but in this scenario, the creation is disabled (set to off) since Terraform is used to manage log groups.
The log_stream_prefix sets a prefix for the log streams created in CloudWatch, helping organize and identify the log entries within the stream.

[OUTPUT]
   Name cloudwatch_logs
   Match applications.*
   region ${AWS_REGION} 
   log_group_name /aws/eks/${CLUSTER_NAME}/workloads
   log_stream_prefix from-k8-fluent-bit-
   auto_create_group off

Once you deploy the Helm chart you can check the CloudWatch service, if everything is working you should see some stream created, in this case out prefix is from-k8-fluent-bit-

and the log entry

Adding a filter

Filters in Fluent Bit allow you to enrich the data being collected. For instance, the Kubernetes Filter adds valuable metadata to log entries, such as namespace, pod_name, host, and more.

Here are some key points about the filter configuration:

The Tag from the input configuration is reused here to extract information like pod_name, namespace, and other relevant metadata.
The Kube_URL points to the Kubernetes API server, which Fluent Bit queries to obtain metadata about the pods involved in the logs. The path for the token and certificate is specified in Kube_CA_File and Kube_Token_File.
You can configure the filter to include annotations and labels from the pods in the log entries.

Note: Be cautious about Fluent Bit querying the API server for metadata. In clusters with a high number of resources, this can put an additional load on the API server. One optimization is to retrieve pod metadata from the node’s kubelet instead of the kube-apiserver, but this requires enabling hostNetwork in the DaemonSet

[FILTER]
   Name   kubernetes
   Match  applications.*
   Kube_URL      https://kubernetes.default.svc:443
   Kube_CA_File       /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
   Kube_Token_File 
     /var/run/secrets/kubernetes.io/serviceaccount/token
   Kube_Tag_Prefix     application.var.log.containers.
   Merge_Log           On
   Merge_Log_Key       log_processed
   K8S-Logging.Parser  On
   K8S-Logging.Exclude Off
   Labels              On
   Annotations         Off
   Buffer_Size         0

After applying this filter the logs should have pods metadata

Option 2. Use the amazon-cloudwatch-observability add-on

Container Insights can be used to collect, aggregate, and summarize both metrics and logs. If you plan to enable this in your EKS cluster, the Amazon CloudWatch Observability add-on installs the necessary resources to achieve this.

At a high level, the add-on installs two key components:

A CloudWatch agent to collect metrics.
Fluent-Bit to collect logs, using the AWS for Fluent-bit container image.

Both components are deployed as DaemonSets.

The add-on can be installed via Terraform or by using a Helm-chart, Regardless of the method, you'll need to create an IAM role for the service account cloudwatch-agent in the amazon-cloudwatch namespace.

resource "aws_eks_addon" "cloudwatch" {
  cluster_name                = aws_eks_cluster.kube_cluster.name
  addon_name                  = "amazon-cloudwatch-observability"
  addon_version               = ""v2.1.2-eksbuild.1""
  service_account_role_arn    = aws_iam_role.cloudwatch_role.arn 
  resolve_conflicts_on_update = "OVERWRITE"
}

The add-on creates several resources in the cluster, some of which you may not need. For example, if you list the DaemonSets in the amazon-cloudwatch namespace, you'll notice seven DaemonSets, some of which might have 0 replicas. While these resources may not be actively used, they still exist in your cluster and can create some noise.

You can customize the add-on configurations to suit your needs. For example, you can disable Fluent Bit logs for Accelerated Compute monitoring or skip collecting NVIDIA GPU metrics.

resource "aws_eks_addon" "cloudwatch" {
  cluster_name                = aws_eks_cluster.kube_cluster.name
  addon_name                  = "amazon-cloudwatch-observability"
  addon_version               = ""v2.1.2-eksbuild.1""
  service_account_role_arn    = aws_iam_role.cloudwatch_role.arn 
  resolve_conflicts_on_update = "OVERWRITE"
  configuration_values = jsonencode({
    containerLogs = {
        enabled = true
    },    
    agent = {
      config = {
        logs = {
          metrics_collected = {
            application_signals = {},
            kubernetes = {
              "enhanced_container_insights": true
              "accelerated_compute_metrics": false
            }}}}}
  })
}

By default, the add-on creates four CloudWatch log groups. The following image, taken from the AWS documentation, explains the naming structure of the log groups and the type of data each group stores.

To change expiration days and names for the groups is better to use the Helm chart instead of the Terraform code to install the add-on. You can do this by modifying the fluent-bit outputs.

The last log group is named performance and stores metrics collected by the CloudWatch agent, such as the number of running pods, CPU usage, and memory metrics.

Bonus: Cluster dashboard

As mentioned earlier, the CloudWatch add-on collects, aggregates, and summarizes metrics. Once the add-on is installed, AWS automatically generates a dashboard that provides useful insights and metrics for your cluster.

You can watch metric per pod

It also generates a visual map that organizes Kubernetes resources by namespace.

Using Terraform to push files to Git Repo for GitOps

Daniel German Rivera — Thu, 11 Jul 2024 21:32:30 +0000

Note: This post was updated because I incorrectly stated that the GitHub Terraform provider didn't delete files in the remote repository when they were removed from the Terraform code. The issue arose because I was using GitHub Actions to run the plan and apply steps, but the GitHub token was not propagating correctly during the plan step. This caused Terraform to fail to delete the files in the remote repository and resulted in multiple commits with each execution. I apologize for any confusion this may have caused and have edited the article to provide accurate information.

I have been working on a personal project named Smart-cash to improve some skills and learn new ones.

In this article, I will share my thoughts about using Terraform in the GitOps process, specifically to create the manifest and push it to the Git repo.

The basics

GitOps relies on a Git repository as the single source of truth. New commits imply infrastructure and application updates.

Imagine a Git repository where you push all the manifests of the Kubernetes resources you want to create in your cluster. These are pulled by a tool or script that runs a "kubectl apply", creates the resources, and checks the Git repo for new changes to apply. This, at a high level, is GitOps.

Setting up the scenario

For this case, the K8 cluster will run in AWS EKS, and Terraform is being used as an IaC tool.

A basic cluster can be created using Terraform. You can check an example here.

FluxCD installation can be done using the official documentation or you can check this.

I will not explain some Flux concepts like sources and Kustomizations; you can check that in the links shared previously.

Creating the YAML files

Let's say that we want to create a namespace for the development environment, we can use the following YAML:

apiVersion: v1
kind: Namespace
metadata:
  name: develop
  labels:
    test: true

We can push this file to GitHub and wait for FluxCD to do the magic.

Now let's say that we want to create a service account and associate it with an AWS IAM role, the YAML can be:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-test-develop
  annotations:  
    eks.amazonaws.com/role-arn: arn:aws:iam::12345678910:role/TEST

This looks easy but what happens if we have multiple environments or if We don't yet know the ARN of the role because this is part of our IaC?

Here is where Terraform gives us a hand.

You can create something like a template for the manifest and some variables that you can specify with Terraform. The two manifests would look like:

apiVersion: v1
kind: Namespace
metadata:
  name: ${ENVIRONMENT}
  labels:
    test: true

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-test-${ENVIRONMENT}
  annotations:  
    eks.amazonaws.com/role-arn: ${ROLE_ARN}

Notice the ${ENVIRONMENT} and ${ROLE_ARN} variables added.

We can use the Terraform GitHub provider to push the file to the repository. Let's check the following code to push the service account:

resource "github_repository_file" "sa-test" {
  repository          = data.github_repository.flux-gitops.name
  branch              = main
  file                = "./manifest/sa-manifest.yaml"
  content = templatefile(
    "sa-manifest.yaml",
    {
      ENVIRONMENT = var.environment
      ROLE_ARN = aws_iam_role.arn
    }
  )
  commit_message      = "Terraform"
  commit_author       = "terraform"
  commit_email        = "example@example"
  overwrite_on_create = true
}

The arguments repository and branch allow us to specify the remote repo and the branch where we want to push the file. The file argument is the location in the remote repository where we want to put the file.

The content argument is where we pass the values to the variables created in the template, in this case ENVIRONMENT and ROLE_ARN, the values are a terraform variable and the reference to a Terraform resource that creates the role.

overwrite_on_create argument is needed because if you run Terraform again, it will show an error because the file already exists in the repo.

Pros

Pushing the manifests using Terraform avoids the manual tasks of committing and pushing them, allowing us to automate more steps.
We can integrate this process into our pipeline, so a full environment can be ready when the pipeline finishes.
Terraform count can be used when there are many manifests to push, avoiding repetitive code.

Smart-Cash project -Adding monitoring to EKS using Prometheus operator

Daniel German Rivera — Thu, 30 Nov 2023 15:54:00 +0000

Previous articles showed how to build the EKS Infrastructure in AWS and how to install FluxCD to implement GitOps practices, This article is focused on explaining the steps taken to install the Prometheus Operator(using Helm) and Grafana for monitoring.

Source Code

The source code for this project can be found here, also a GitOps repository has been created to store the Yaml files that FluxCD uses and apply to the EKS cluster.

Prometheus operator

The Prometheus Operator provides Kubernetes native deployment and management of Prometheus and related monitoring components.

The Prometheus operator defines Kubernetes Custom Resources and controllers that facilitate installing Prometheus. The community has developed alternative options such as kube-Prometheus and kube-prometheus-stack to install the components to monitor Kubernetes.

Prometheus Operator, kube-prometheus and kube-prometheus-stack

The project repository for Prometheus-operator can be found here, The repo defines the CRDs and the controller. You can follow this documentation for the installation. which will require the creation of metrics exporters, node exporters, scrape configurations, etc.

On the other hand, the Kube-prometheus project provides documentation and scripts to operate end-to-end Kubernetes cluster monitoring using the Prometheus Operator, making easier the process of monitoring the Kubernetes cluster.

kube-prometheus-stack is a Helm chart that contains several components to monitor the Kubernetes cluster, along with Grafana dashboards to visualize the data. This option will be used in this article.

Installing kube-prometheus-stack Helm chart

In previous articles, FluxCD was installed in EKS cluster to implement GitOps, the following flux source will now be added.


 YAML
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: helm-repo-prometheus
  namespace: flux-system
spec:
  interval: 10m0s
  url: https://prometheus-community.github.io/helm-charts

Also, a Flux Helm release is added


 YAML
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: prometheus
  namespace: monitoring
spec:
  interval: 10m0s
  chart:
    spec:
      chart: kube-prometheus-stack
      sourceRef:
        kind: HelmRepository
        name: helm-repo-prometheus
        namespace: flux-system
  values:
    defaultRules:
      rules:
        etcd: false
        kubeSchedulerAlerting: false
        kubeSchedulerRecording: false
        windows: false
    prometheus:
      prometheusSpec:
        storageSpec:
            volumeClaimTemplate:
              spec:
                storageClassName: aws-ebs-gp2
                accessModes: ["ReadWriteOnce"]
                resources:
                  requests:
                    storage: 40Gi

If you examine the values for the chart, by default, it installs rules to monitor etcd and some other control plane components. However, in this case, that is not necessary due to EKS limiting access to certain control-plane components.

By default, Prometheus uses local storage to store data. To enable persistent storage, an EBS volume can be added. In this scenario, the EBS CSI driver is employed, and a storage class is defined to manage the integration with Prometheus.

Once the manifests are ready, you can push them to the GitOps repo( here for this case), and wait for Flux to handle the installation process in the cluster.

You can check the cluster by looking for the resources created, notice that for this case, everything will be placed in the monitoring namespace.

kubectl get crd

Additionally, certain deployments and services should have been created in the monitoring namespace.

kubectl get pods -n monitoring

Let's look at the Prometheus server console, you can expose the service by an ingress or using port-forward.

kubectl port-forward service/prometheus-kube-prometheus-prometheus 3001:9090 -n monitoring

The previous command will expose the Prometheus service in localhost:3001, you can go directly to the targets and you should see some targets created automatically, as well as the metrics and the services discovered by the server.

This is useful because you don't need to configure the targets to monitor K8 and node metrics, the Helm chart does this for you. For instance, a simple example is just to check the number of pods created in the default namespace, you can run this PromQL query.

count(kube_pod_created{namespace="default"})

Grafana Dashboards

Helm chart also installs Grafana and configures some useful dashboards that you can use, if you list the services and pods you will see some resources related to Grafana. You can expose the Grafana service or create an ingress for it.

Creating nginx ingress for Grafana

Nginx-ingress is utilized and installed using Helm. You can add the following Flux source and the Helm release to the GitOps repo. Check the GitOps repo for this project here and use it as a model.

Helm source



apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: helm-repo-nginx-ingress
  namespace: flux-system
spec:
  interval: 10m0s
  type: oci
  url: oci://ghcr.io/nginxinc/charts

Use this yaml to install the chart, in this case AWS Network Load Balancer is used, this is done through the annotation specified in the values for the chart.


 Yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  interval: 10m0s
  chart:
    spec:
      chart: nginx-ingress
      version: 1.0.2
      sourceRef:
        kind: HelmRepository
        name: helm-repo-nginx-ingress
        namespace: flux-system
  values:
    controller:
      service:
        annotations: 
          service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

Installing Cert-managet to support SSL

This article will not dig into details about cert-manager concepts.

In order to support SSL in the EKS cluster cert-manager will be used, cert-manager adds certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing and using those certificates.

Cert-manager uses Kubernetes CRD, to install them you can run:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.crds.yaml

This also can be added to the GitOps repo(check here) and given to Flux to handle it.

When the CRDs are ready, you can install cert-manager, in this case a Helm chart will be used, this also will be added in GitOps repo.



apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: cert-manager
  namespace: cert-manager
spec:
  interval: 10m0s
  chart:
    spec:
      chart: cert-manager
      version: 1.13.2
      sourceRef:
        kind: HelmRepository
        name: helm-cert-manager
        namespace: flux-system
  values:
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/cert-manager-us-west-2
    securityContext:
      fsGroup: 1001
    extraArgs:
      - --issuer-ambient-credentials

Finally and cert-manager ClusterIssuer is added, in this case the domain will be validated in AWS Route53, this is done through the IAM role passed in the previous YAML file for the Helm chart.



apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: example-letsencrypt2
spec:
  acme:
    email: notreply@example.info
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: example-issuer-account-key
    solvers:
    - selector:
        dnsZones:
          - "example.info"
      dns01:
        route53:
          region: us-west-2

Once cert-manager and nginx-ingress are installed you can create an ingress for Grafana. The following manifest has been added to the GitOps repo.


 YAML
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: monitoring
  annotations:
    cert-manager.io/cluster-issuer: example-letsencrypt2
spec:
  ingressClassName: nginx
  rules:
  - host: monitoring.example.info
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-grafana
            port:
              number: 80
  tls:
   - hosts:
     - monitoring.example.info
     secretName: example-issuer-account-ingress-key2

With this installed you can browser and access Grafana, you should see some dashboards already created.

For instance, the Kubernetes/API server dashboard is pre-configured, also you can also use third-party dashboards.

Smart-Cash Project - GitOps with FluxCD

Daniel German Rivera — Sat, 04 Nov 2023 21:54:01 +0000

In a previous article I mentioned the idea behind this project that I named SmartCash. I began building the terraform code for the infrastructure in AWS and the pipeline to deploy it.

In this article, I will introduce FluxCD as a GitOps tool and demonstrate its usage.

Source code

A new release has been created in the smart-cash repository for the project. v1.1.0 version will be used, you can check the repository here.

Additionally, a new repository will be created to store the K8 manifest that will be synced with the EKS cluster using FluxCD, you can view the repo here.

A quick introduction to GitOps

GitOps is an operational model for cloud-native architectures, it relies on a Git repository as the single source of truth. New commits imply infrastructure and application updates.

OpenGitOps group has defined 5 principles, and while I won't delve into them, here, you can read more. If you take a look at those principles you will see that they are, in some sense related to some Kubernetes concepts.

A great book to gain a better understanding of GitOps history and concepts is The Path to GitOps.

In summary, GitOps is centered around using a Git repository for defining and managing both infrastructure and application configurations through a Git-based workflow.

What is FluxCD

FluxCD is an open-source GitOps operator for Kubernetes, you can declaratively define the desired state of your infrastructure and configurations in a Git repository. Flux monitors the repository and applies updates to the Kubernetes cluster when new changes arrive.

Flux started as a monolith but in v2 it was broken up into individual components called GitOps Toolkit, this refers collection of specialized tools, Flux Controllers, composable APIs, and reusable Go packages available under the fluxcd GitHub organization.

Core concepts and toolkit components are described here.

Installing FluxCD in the cluster

FluxCD installation can be done by Flux CLI, the most straightforward method can be done by the flux bootstrap command, this deploys the Flux controllers on the K8 cluster and configures them to synchronize the cluster to the Git repository, if the Git Repo doesn't exist, the bootstrap command will create it.

To incorporate FluxCD installation into this project a new bash script has been added into the repository that contains the terraform code, this bash script will be execute by terraform as a null resource.

#/bin/bash

## Configure Cluster Credentials

# $1 = CLUSTER_NAME
# $2 = AWS_REGION
# $3 = GH_USER_NAME
# $4 = FLUX_REPO_NAME

echo "---------->  get eks credentials"
aws eks update-kubeconfig --name $1  --region $2

## validate if flux is installed

flux_installed=$(kubectl api-resources | grep flux)
if [ -z "$flux_installed" ]; then
  echo "---------->  flux is not installed"

  ### install flux

  echo "---------->  installing flux cli"

  curl -s https://fluxcd.io/install.sh | sudo bash

  echo "---------->  run flux bootstrap"
  flux bootstrap github \
    --owner=$3 \
    --repository=$4 \
    --path="clusters/$1/bootstrap" \
    --branch=main \
    --personal
else
  echo "---------->  flux is installed"
fi

The flux bootstrap github command deploys the Flux controllers on the K8 cluster and configures the controllers to synchronize the Git repo with the cluster. This is done by some K8 manifests that are created and pushed to the repo in the path passed in the command.

It's worth noting that some env variables like FLUX_REPO_NAME, and GH_USER_NAME are used by the bash script, these variables are passed as an argument in the bash script execution.

Adding FluxCD bootstrap script to terraform code

The bash script will be executed in the GH workflow template created to deploy the infrastructure, the following job is added to the Workflow template.

#### bash script arguments
  # $1 = CLUSTER_NAME
  # $2 = AWS_REGION
  # $3 = GH_USER_NAME
  # $4 = FLUX_REPO_NAME

resource "null_resource" "bootstrap-flux" {
  depends_on          = [module.eks_cluster]
  provisioner "local-exec" {
    command = <<EOF
    ./scripts/bootstrap-flux.sh ${local.cluster_name}  ${var.region} ${local.gh_username} ${data.github_repository.flux-gitops.name}
    EOF
  }
  triggers = {
    cluster_oidc = module.eks_cluster.cluster_oidc
    created_at   = module.eks_cluster.created_at
  }

}

Notice that the GITHUB_TOKEN variable is passed directly in the Github job.

Once the workflow is ready you can push it to the repo and see how terraform will create all the infra and after EKS cluster creation will execute the bash script.

You can run flux check command locally to validate the status of the installation(you should have access to the cluster in your local env)

If you take a look at the above image you will see that the Source Controller is deployed, Source Controller enables seamless integration of various Git repositories with your Kubernetes cluster. Think of the Source Controller as an interface to connect with GitRepositories, OCIRepository, HelmRepository, and Bucket resources.

✔ source-controller: deployment ready

The bootstrap command will create a flux source and associate it to the repo passed in the command, to validate this you can list the git sources created and you will see the one, for now.

flux get sources git

and you can see the K8 CDRs created

kubectl get crds | grep flux

Structuring the Git repository

There are different strategies to structure the GitOps repository, for this scenario, a mono-repo strategy is used and kustomize will be used to manage the K8 manifest for the application.

./clusters: contains all the cluster associated with the project, cluster for each environment or region should be placed here.
./clusters/smart-cash-develop/bootstrap: Yaml files created by fluxcd installation, also there is a file name core-kustomization.yaml that points to a core folder that manages the manifests.
./clusters/smart-cash-develop/core: Contains the main manifest for the project, manifest like FluxSources, and also kustomization files. Here will be placed the kustomization file for each microservice that will be created.
./clusters/smart-cash-develop/core: Manifests that create common resources for the cluster like namespaces, ingress, storage-classes, etc.
Manifests: This contains subfolders that contain the YAML files for each microservices.

├── clusters
    └── smart-cash-develop
        |── bootstrap
        |── common
        |   |── ingress-namespace.yaml
        |   └── namespaces.yaml
        |── core
        |   |── common-kustomize.yaml
        |   └── helm-cert-manager.yaml
        └── manifests
            └── app1
                |── base
                |   |── kustomization.yaml
                |   └── deployment.yaml
                └── overlays
                    |── develop
                    |   └── kustomization.yaml
                    └── production
                        └── kustomization.yaml

Adding resources to the cluster

Let's create a K8 namespace to be used for an nginx-ingress. The manifest for this can be placed in the common folder. A FluxCD Kustomization can be added to synchronize the contents of this folder with the K8 cluster.

The following is the Flux Kustomization that reconciles the Kubernetes manifests located at the path ./common in the Git repository .

Note: This file can be added in clusters/smart-cash-develop folder, FluxCD will automatically create the Kustomization resource because this path was specified in the bootstrap command, and Flux created a Kustomization to synchronize it.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: smartcash-common
  namespace: flux-system
spec:
  interval: 5m
  targetNamespace: default
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: "./kustomize"
  prune: true

interval: The period at which the Kustomization is reconciled.
sourceRef: refers to the Source object with the required Artifacts, in this case, our GitOps repository.
prune:: When is true, if previously applied objects are missing from the current revision, these objects are deleted from the cluster

Once you push the Yaml file to the GitOps repo, Flux will create the resources in the cluster. You can validate running:

kubectl get kustomization -n flux-system

The previous steps have created the FluxCD Kustomization to sync the common folder with the cluster. Now, a Kustomize file needs to be added to specify which resource to create.

Don't confuse the FluxCD Kustomization file with the K8 configuration management Kustomize. FluxCD will look for the Kustomize file in the common folder.

Let's create and push the following files in the common folder.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ns-nginx-ingress.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: nginx-ingress

You can wait for the flux reconciliation or force it using the following command:

flux reconcile kustomization smartcash-common

If the process was successful you should see the nginx-ingress namespace.

Troubleshooting

To validate the status of the reconciliation you can use the following command:

flux get kustomization smartcash-common

For instance, a mistake in the name of the YAML files caused this error, which was visible in the output of the flux command.

If you want more details you can check the K8 CDRs using:

kubectl describe kustomization smartcash-common -n flux-system

Creating a Helm release for nginx-ingress

The Flux Helm Controller will be used to install the ingress. The Helm Controller is a Kubernetes operator that enables the management of Helm chart releases.

A FluxCD source for Helm needs to be added. This can be accomplished by using the following manifest, which should be placed in clusters/smart-cash-develop.

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: helm-repo-nginx-ingress
  namespace: flux-system
spec:
  interval: 5m0s
  type: oci
  url: oci://ghcr.io/nginxinc/charts

This source fetches the Helm OCI repository oci://ghcr.io/nginxinc/charts every 5 minutes, and the artifact is stored and updated each time new updates are done to the repository.

After creating the Helm source, you can proceed to create the Helm release. This release specifies the chart to install in the cluster, with the chart being fetched from the source already created. The following manifest can be used.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  interval: 10m0s
  chart:
    spec:
      chart: nginx-ingress
      version: 0.17.1
      sourceRef:
        kind: HelmRepository
        name: helm-repo-nginx-ingress
        namespace: flux-system

To delegate the creation of the HelmRelease task to flux, this file can be added to the common folder and in the Kustomize file as well.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ns-nginx-ingress.yaml
- nginx-ingress-helm.yaml

After updating and pushing the files, you can validate the creation of the Helm Release and nginx-ingress resources.

flux get helmreleases -n nginx-ingress

Up to this point, we've covered the second phase of this project. In the upcoming articles, you'll delve into the implementation of various other tools and continue building the project.

If you have any feedback or suggestions, please feel free to reach out to me on LinkedIn.

Smart-Cash Project - AWS Infrastructure - Terraform and GitHub Actions

Daniel German Rivera — Wed, 25 Oct 2023 22:12:36 +0000

The journey to learn a new tool can be a little tricky, watching videos and reading some blogs can be an option, but watching and reading can not be enough for everyone, personally I need a hands-on approach for effective learning.

That motivation drove me to embark on a personal project where I could implement the tools I had been using and those I wanted to explore. to initiate this journey I decided to create an application that helped me to follow my expenses, it could be something trivial but I needed a "business case" to begin building a solution.

The general idea is to have an application that helps you track the monthly expenses and filter for a particular category to see where the money is going.

Let's start building the initial infrastructure needed for the application.

Here you can find the GitOps implementation(Part 2 of the project).

Source Code

The version of the code used in this article can be found here. only specific sections of the code are included here to provide explanations, as this approach avoids the need to paste the entire code.

Architecture

The image below shows the first version of the architecture to use, I chose Kubernetes for this project as it allows me to explore some tools for K8 and improve some skills to present a K8 certification.

IaC

Terraform is the tool for IaC, while I won't delve into a detailed terraform code explanation, I will provide some key highlights:

Networking

A third-party TF module is used. to save some aws costs the NAT gateways have been disabled.

EKS

A TF module has been developed to deploy the resources needed for an EKS cluster. I created manually an IAM user and passed it as a variable(var.userRoleARN) to the EKS module. This internally runs an eksctl command to add it to the RBAC configs. This user will be used to operate the cluster locally.

resource "null_resource" "iam-role-cluster-access" {
  provisioner "local-exec" {
    command = <<EOF
      curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
      /tmp/eksctl version
      /tmp/eksctl create iamidentitymapping --cluster ${local.eksClusterName} --region=${var.region} --arn ${var.userRoleARN} --group system:masters --username "AWSAdministratorAccess:{{SessionName}}"
    EOF
  }
  depends_on = [
    aws_eks_cluster.kube_cluster,
    aws_eks_node_group.worker-node-group
  ]
}

EKS worker nodes will run in public subnets, this is because I have disabled the NAT GW to save some costs.

Infrastructure Pipeline

Branch strategy

I will follow a common branch strategy as the following image shows. The main branch is associated with the production environment.

GitHub Actions

GitHub actions will be used to implement the pipeline to deploy the infrastructure in AWS, if you are not familiar with GitHub actions terminology you can check the documentation.

You can find the YAML files in the .github folder, which also contains other 3 subfolders, let's explore them in detail.

actions folder

This folder contains the GitHub composite Actions to use in the workflows, you can think of an action as a template that defines the task to execute(jobs). Now let's review the terraform-plan action.

The first part defines the name and the inputs for the action, in this case, I am just defining the working directory where TF is placed as an input.

name: 'Terraform Plan'
description: 'Running Terraform plan'

inputs:
  WORKING_DIRECTORY:
    description: 'directory where the tf code is'
    required: false
    default: '/infra/terraform'

The second part defines the tasks to execute, this action will start installing Terraform.

runs:
  using: "composite"
  steps:
      - name: Terraform install
        id: 'install-terraform'
        uses: hashicorp/setup-terraform@v2
        with: 
         terraform_version: '${{ env.TERRAFORM_VERSION }}'
      - name: Validate terraform version
        id: validate-tf-version
        run: terraform version
        shell: bash

Notice that the step Terraform install is using an external GH action, hashicorp/setup-terraform@v2, the version of this action is specified after the @. This action is available in the GH actions marketplace.

The next step is to run terraform init but I won't delve into detail, therefore, let's proceed to the final two steps.

The step Run terraform plan runs the TF plan command passing some variables that are defined in the workflow definition, the plan generated is saved in a file named with the GH actions run id.

Finally, the plan generated in the previous step is published as an artifact, that is used for the action created for the TF apply process.

      - name: Run terraform plan
        id: terraform-plan
        run: | 
            terraform plan \
            -input=false \
            -var 'region=${{ env.AWS_REGION }}' \
            -var 'environment=${{ env.ENVIRONMENT }}' \
            -var 'project_name=${{ env.PROJECT_NAME }}' \
            -out ${{ github.run_id }}.tfplan
        shell: bash
        working-directory: '.${{ inputs.WORKING_DIRECTORY }}'

      - name: Publish Artifact
        uses: actions/upload-artifact@v3
        with:
          name: tf-plan
          path: '${{ github.workspace }}${{ inputs.WORKING_DIRECTORY }}/${{ github.run_id }}.tfplan'

jobs

This folder stores some bash scripts used in the pipelines to perform specific tasks, currently just one script is stored here, and its purpose is to create the S3 bucket and the DynamoDB tables used for the TF state.

The script runs some AWS CLI commands to validate if the S3 bucket and DynamoDB table exist, if not the resources are created.

A composite action has been created to execute this script, you can find it with the name terraform-backend.

name: 'Terraform backend set-up'
description: 'set-up terraform plan'
runs:
  using: "composite"
  steps:
    - name: Config tf backend
      id: tf-backend
      run: ./terraform-backend.sh
      shell: bash
      working-directory: .github/workflows

Workflows

Workflows define the triggers(commits, tags, branches...) for the pipeline and also specify the process to execute(jobs), inside the workflow you can use the composite actions already defined.

Workflow Template

I have created a workflow template that defines the common tasks for all the environments, but this does not define the triggers. let's take a look at the template.

name: terraform deploy template
on:
  workflow_call:
    inputs:
      AWS_REGION:
        description: 'aws region where the resources will be deployed'
        required: true
        type: string
     secrets: 
      AWS_ACCOUNT_NUMBER:
        required: true

The initial section defines the inputs and secrets for the workflow, the main difference between inputs and secrets is that Github hides the value of the secret in the workflow logs.

The second part of the template defines some common environment variables.

env:
  ENVIRONMENT: ${{ inputs.ENVIRONMENT }}
  AWS_REGION: ${{ inputs.AWS_REGION }}
  PROJECT_NAME: ${{ inputs.PROJECT_NAME }}
  TERRAFORM_VERSION: ${{ inputs.TERRAFORM_VERSION }}
  AWS_IAM_ROLE_GH: 'GitHubAction-AssumeRoleWithAction'

You can observe that the value for the env variable AWS_REGION is set to the value passed by the input inputs.AWS_REGION defined earlier. Why it is done? let's review one snippet of code from the composite action for the terraform plan to gain a better understanding.

- name: Run terraform plan
        id: terraform-plan
        run: | 
            terraform plan \
            -input=false \
            -var 'project_name=${{ env.PROJECT_NAME }}' \

As you can see I'm using the env variable PROJECT_NAME and passing it as a TF variable, this is possible because in the workflow I defined the value for the variable, and this is passed down in the runner. You can use inputs here but you would need to define the same input in the composite action.

The last part of the template defines the jobs to execute


jobs:
## Execute bash script that  create s3 bucket and dynamodb table for Terraform backend
  set-up-terraform-backend:
    runs-on: ubuntu-latest
    steps:  
      - name: checkout-repo
        uses: actions/checkout@v4
      - name: configure aws credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: 'arn:aws:iam::${{ secrets.AWS_ACCOUNT_NUMBER }}:role/${{ env.AWS_IAM_ROLE_GH }}' 
          role-session-name: GitHub_to_AWS_via_FederatedOIDC
          aws-region: ${{ env.AWS_REGION }}
      - name: config tf backend
        id: tf-backend
        run: ./terraform-backend.sh
        working-directory: .github/jobs/

The provided code shows the job to set up the Terraform backend, this job is executed in a Ubuntu runner that is defined by runs-on.

Let's check the steps executed for the job.

The first step is to make a checkout of the repo into the runner, this is done by an external composite action.
To execute Terraform the job needs access to AWS, this is done by an external composite action, to avoid pass Access and Secret keys OpenID Connect (OIDC) will be used, which allows GitHub Actions workflows to access AWS. You need to create an IAM IdP in your AWS account and associate it with an IAM role, this role must contain the permissions that Terraform needs to run properly. The details for the configuration can be checked here.
Once the AWS credentials have been configured you can call the composite action created to set up the Teraform backend.

The other jobs follow a similar pattern but they execute the composite actions for the Terraform plan and apply.

Using the template

Once the template is ready you can create the workflows for each environment. let's review the workflow for the develop environment.

name: Terraform infra workflow DEVELOP
run-name: terraform-deploy-DEVELOP

on: 
  push:
    branches:
      - develop
    paths:
      - "infra/**"
  pull_request:
    branches:
      - develop

permissions:
  id-token: write # This is required for requesting the JWT
  contents: read  # This is required for actions/checkout

defaults:
  run:
    shell: bash
    working-directory: ./infra/terraform

jobs:
  ### Makes a call to the workflow template defined to execute terraform, in this case, the variables define the develop environment
  terraform-deploy:
    uses: danielrive/smart-cash/.github/workflows/run-terraform-template.yaml@develop
    with: 
      AWS_REGION: 'us-west-2'
      ENVIRONMENT: 'develop'
      PROJECT_NAME: 'smart-cash'
      TERRAFORM_VERSION: '1.4.6'
    secrets:
      AWS_ACCOUNT_NUMBER: ${{ secrets.AWS_ACCOUNT_NUMBER_DEVELOP }}

I have defined two triggers for the workflow, The first trigger is when new updates are pushed to infra folder within the develop branch, and the second trigger is when a new Pull request is open with the develop branch as a base.

The permissions section is necessary to generate a token used to establish the connection with AWS IAM IdP.

In the job definition, you can is where you can utilize the previously created template, you need to specify the path where the template is located and the values for the inputs defined in the template.

You can create other workflows for other environments and pass the respective values for inputs.

Up to this point, you've covered the first phase of this project. In the upcoming articles, you'll delve into the implementation of various other tools and continue building the project.

Containers - entre historia y runtimes

Daniel German Rivera — Wed, 26 Apr 2023 22:50:09 +0000

Estudiando kubernetes gasté un tiempo considerable intentando entender muchos conceptos, por ejemplo, por todo lado se habla de OCI compliant, buscas OCI y te lleva a runtime-spec, buscas runtimes y te lleva a containerd, runc, image-spec, cgroups, namespaces, etc; puedes pasar días buscando, y mucho más cuando eres del tipo de persona que quiere entender a fondo cómo funcionan las cosas.

Motivado por lo anterior, me decidí a escribir este post con la idea de compartir los conceptos que logré adquirir y que me han servido para entender varias cosas del gran mundo de los containers, en algunas cosas no voy a tan bajo nivel ya que hay muchos conceptos que todavía desconozco y puedo decir cosas equiviocadas.

Lo básico

Iniciemos entendiendo un poco la idea detrás de los containers.

Containers tienen como objetivo crear un ambiente virtual aislado el cual se pueda distribuir y desplegar fácilmente. Dentro del container pueden correr diferentes procesos los cuales deben estar aislados de otros corriendo en el host. El kernel de linux ofrece distintas funcionalidades que permiten la creación de estos ambientes. Hay dos componentes principales que quizás son el core de todos los containers.

Linux namespaces

Linux namespaces nos permite crear ambientes virtuales y aislados, estos particionan recursos del kernel y hacen que sean visibles solo para los procesos que corren dentro del namespace, pero no para procesos externos. En otras palabras, namespaces nos facilitan el aislamiento entre procesos.

¿Qué recursos se pueden particionar?, bueno esto va a depender del tipo de namespace que se este usando, por ejemplo, network namespaces nos permite encapsular los recursos relacionados con networking, como interfaces, tablas de rutas, etc. De esta forma podemos crear una red virtual dentro de nuestro namespace.

Este post explica un poco más en detalle los namespaces.

cgroups

Recordemos que el Kernel de Linux es la interfaz principal entre el hardware y los procesos, permitiendo la comunicación entre estos dos y ayudando a la gestión de recursos, por ejemplo, puede terminar procesos que consuman demasiada memoria para evitar afectar el sistema operativo. Adicionalmente pueden controlar qué procesos pueden consumir cierta cantidad de recursos.

cgroups es una funcionalidad del Kernel de Linux que permite organizar jerárquicamente procesos y distribuir recursos(cpu, memoria, networking, storage) dentro de dicha jerarquía.

Configurar cgroups puede ser un poco complejo, en mi caso estuve leyendo varios post acerca del tema y requiere cierto tiempo para entender por completo su funcionamiento. En esta serie de posts creados por RedHat se habla sobe cgroups y su configuración a través de systemd, pero si se desea entrar en detalle la documentación de Linux puede ser de ayuda.

cgroups y namespaces se convierten en los ingredientes secretos en la creación de containers, namespaces permiten aislamiento a nivel de recursos y cgroups permiten controlar los limites para dichos recursos.

Por suerte hoy en día con una sola linea podemos crear un container, no tenemos que entrar a configurar namespaces ni cgroups.

Veamos un poco de la evolución de los containers y así vamos aclarando ciertas cosas.

Un poco de historia

Docker fue el primero que popularizó los containers, era(o es) común asociar containers directamente con Docker, pero antes ya existía algo llamado LXC(Linux containers), el cual puede entenderse como un proveedor de ambientes virtuales en Linux que usa ciertos componentes del Kernel de Linux para crear ambientes aislados(containers).

LXC se encuentra dentro del user-space, es decir, nosotros interactuamos con LXC y este se encarga de interactuar con los componentes del kernel para permitir la creación de containers. Aqui un video en donde se puede ver LXC en acción.

Nota: Antes de LXC ya se habían desarrollado otros alternativas para la creación de containers como OpenVZ y Linux Vserver. LXC es mencionado inicialmente ya que es lo más cercano a Docker que es el software con el que muchos iniciamos interactuando con containers.

La llegada de Docker

Docker empaquetó LXC en una herramienta que facilitaba más la creación de containers. Al ganar popularidad se crearon mejoras y unos meses después Docker lanzó libcontainer el cual está escrito en Golang y básicamente reemplazaba LXC.

Docker se enfocó más en la creación de containers optimizados para el despliegue de aplicaciones mejorando la portabilidad. Este post explica más detalladamente las diferencias entre LXC y Docker.

Definiendo un estándar para containers

Como alternativa a Docker, empezaron a surgir otras opciones,CoreOS por su parte lanzó rkt(2014) proponiendo mejores de seguridad, CoreOS argumentaba que Docker había sido construido como un monolito el cual corría como root en el host, abriendo posibilidades a comprometer todo el host en el caso de un ataque.

rkt usa appc(open source container) con el fin de mejorar la operabilidad, appc tiene como propósito crear un estándar general para crear containers buscando ser vendor-independent y OS-independent.

Otras iniciativas empezaron a surgir debido a la alta popularidad de los containers y debido a esto, en 2015 se crea OCI(Open Container Initiative) para definir un estandar para containers(runtimes e imagenes).

OCI Runtime spec

Runtime spec define la configuración(archivo JSON), ambiente y ciclo de vida de un container. Las configuraciones son definidas en un archivo llamado config.json, el cual contiene la metadata necesaria para la ejecución del container, este archivo es definido de acuerdo a plataforma a usar(windows, linux, solaris, etc).

otro concepto a destacar es el filesystem bundle, este es un grupo de archivos con la data y metadata para correr un container. Los principales archivos que deben contener son, el config.json mencionado anteriormente y el rootfs(linux file system), este filesystem bundle se genera a través del container image.

Todas las especificaciones para el container runtime son descritas aqui.

OCI Image spec

Docker en sus inicios ya había definido las especificaciones para la creación de imágenesImage Manifest 2 Schema Version 2, al ser el más popular, OCI partió de este para crear un estándar más general, que no estuviera asociado a un vendor en específico. Image spec define como construir y empaquetar container images, personalmente no he entendido del todo el funcionamiento pero aquí está la url del repo y un blog-post que contienen mayor información.

Haciendo uso del Image spec, se puede crear un container image que puede ser ejecutada por cualquier OCI Runtime, esto quiere decir que a través del Image spec se puede generar el filesystem bundle, el cual es usado por el runtime para la creación y ejecución del container.

The Runtime Specification outlines how to run a "filesystem bundle" that is unpacked on disk. At a high-level an OCI implementation would download an OCI Image then unpack that image into an OCI Runtime filesystem bundle. At this point the OCI Runtime Bundle would be run by an OCI Runtime.

Container runtimes y Kubernetes

En el 2015 se lanza el primer release de kubernetes, el cual usaba Docker como runtime.

Docker decide dividir el monolito creado. libcontainer es donado a OCI y Docker empieza a trabajar en un proyecto llamado runC, este se puede ver como una herramienta que lee OCI specifications e interactúa con libcontainer para la creación de containers. runC es independiente del Docker Engine y es donado a OCI.

runC es una low-level runtime por lo que también se desarrolla containerd el cual es como una interfaz entre el cliente y runC.

Hasta el momento solo se ha cubierto parte del origen de los container y el origen de algunas herramientas que seguimos viendo hoy en día como runC y conteinerd. En lo que sigue del post trataré de exponer un poco más a fondo las container images al igual que algunas containers runtimes.

Container Images

Antes de entrar a ver las containers runtimes, es importante entender qué es lo que contienen las containers images, para ello vamos a usar Skopeo.

Skopeo permite manipular e inspeccionar container images ya sea para Windows, Linux o MacOs. En este caso vamos a usar Skopeo para obtener "el contenido" de una imagen que se encuentra en DockerHub, esto es muy similar al comando docker export,pero en este caso no vamos a instalar Docker.

copiando images con skopeo

Para instalar skopeo se puede usar snap en ubuntu



sudo snap install skopeo --edge

una vez que finalice la instalación podemos copiar una imagen que se encuentra en DockerHub a nuestro local. En este caso se va a usar la imagen de golang.



sudo skopeo --insecure-policy copy docker://golang:latest  oci://home/ubuntu/example-dev-to/golang-image-v2

Skopeo copia el contenido de la imagen en el destino especificado, en este caso oci:/home/ubuntu/example-dev-to/golang-image-v2. En la imagen se puede ver que se tiene un archivo index.json, oci-layout y un directorio llamado blobs. Esto corresponde a la estructura de archivos definidos por OCI

el index.json se puede entender como un resumen de la imagen, en donde se ve el sistema operativo y la arquitectura, además se especifica la ubicación del image manifest.

El image manifest contiene metadata de la imagen al igual que las especificaciones de cada layer creada.

Revisando el index.json vamos a encontrar lo siguiente:

Se puede ver información acerca del sistema operativo y arquitectura soportados por la imagen. El digest(linea 6) nos indica en que archivo se encuentra el manifest.json.

En el manifest(imagen anterior) se puede ver el digest para el config file y para cada una de las layers que se tienen. El mediaType puede entenderse como el formato de cada archivo, por ejemplo la linea 4 nos dice que el archivo config de formato json se puede identificar con el digest bdba673e96d6e9707e2a724103e8835dbdd11dc81ad0c76c4453066ed8db29fd. Este se puede encontrar en la carpeta blobs y va a lucir como la siguiente imagen.

Este archivo ya contiene más información de la imagen, por ejemplo podemos ver el workdir y algunas variables de entorno.

pasemos ahora a las layers, en el manifest podemos identificar los digest para cada layers, si vemos el media type nos indica que es v1.tar+gzip, en este caso tenemos que descomprimir el contenido de dicho digest, para ello vamos a usar tar

Una vez termine el proceso podemos analizar el resultado, en este caso vamos a tener una serie de directorios que representan el rootfs de la imagen, estos archivos van a hacer parte de un layer en específico. Si observamos la siguente imagen podemos ver que tenemos /home, /etc y /bin, etc, los cuales representan el sistema de archivos de linux(rootfs).

Con esto vemos a alto nivel el contenido de un container image, al final el container runtime es el que se encarga de descomprimir y leer todos estos archivos, el cual va a ser usado para correr el container.

Hasta aquí va la primera parte de este post, en la siguiente veremos un poco m'as los container runtimes.

AWS Event-Bridge and Lambda to copy RDS snapshots to another Region

Daniel German Rivera — Sun, 02 Apr 2023 19:22:31 +0000

A few months ago I was asked to design the DRP process(Multi-Region) for a project that used RDS(PostgreSQL). RDS instances were critical components, these stored PII information. RDS automatically takes snapshots of the instances and you can use them to recreate the instances in case of failure, these snapshots just can be used in the same region but you can share or copy them between accounts and Regions, here some AWS Docs related RDS automatic snapshots.

My initial idea was to create a K8 job to run pg_dump for each RDS instance and then, upload the file to an S3 bucket created in a different region, not too bad but this requires more work to backup and restore. so I decided to copy snapshots between regions, a new challenge appeared here, how to automate that copy? In this post I will show the approach that I follow to solve this, one crucial point to mention here is that for a big number of snapshots, this solution could not be the best due to some limitations that AWS RDS to copy snapshots.

AWS Documentation says:

"You can have up to 20 snapshot copy requests in progress to a single destination Region per account."

This approach uses AWS event-bridge and Lambda to automate the copy process, at summary, even-bridge detects that a new RDS snapshot has been created and triggers a lambda function to copy the snapshot to the other region.

A terraform code was created for this pods and you can check it here

RDS Snapshot

You can configure automated RDS snapshots for your instance, this occurs daily during the backup window that you define in the instance creation.
In this case, the automated RDS snapshot was configured in each instance, this just creates the snapshot in the same account and region where the RDS instance was created.

AWS Event-Bridge

EventBridge is a serverless service that uses events to connect application components together, making it easier for you to build scalable event-driven applications. You can use it to route events from sources such as home-grown applications, AWS services, and third-party software to consumer applications across your organization.

In this case, AWS generates a significant number of events for some services, for RDS you can find the events divided into categories, the following table shows the events for RDS snapshots. when RDS starts an automated snapshot, AWS registers that event. You can find all the events in the AWS documentation.

How to use the events?

Let's start with an important concept in the EventBridge world, An event bus. This a pipeline that receives events, you can configure rules to manipulate the events and specify actions when these came. Events are represented as JSON objects and they all have a similar structure and the same top-level fields. By default, the AWS accounts have a default event bus that receives events from AWS services.

In this case, we can create a rule using the default event-bus.

You can choose the AWS service to use and AWS will show you and JSON with an example of how the event will look.

For our case we can use a simpler event using the EventID for automated snapshots, RDS-EVENT-0091, you can refer to the image shown at the top of the post for more information.


 json
{
  "source": ["aws.rds"],
  "detail-type": ["RDS DB Snapshot Event"],
  "account": ["1234567890"],
  "region": ["us-east-1"],
  "detail": {
     "SourceType": ["SNAPSHOT"],
      "EventID": ["RDS-EVENT-0091"]   
    }
}

With the event-pattern defined, we can specify the lambda function to execute when this event comes to the default event bus.

This means that if an event is received in the default event bus and matches with the pattern specified, that will trigger the lambda and pass a JSON with the event generated, this looks like this:



 json

{

  "version": "0",

  "id": "844e2571-85d4-695f-b930-0153b71dcb42",

  "detail-type": "RDS DB Snapshot Event",

  "source": "aws.rds",

  "account": "123456789012",

  "time": "2018-10-06T12:26:13Z",

  "region": "us-east-1",

  "resources": ["arn:aws:rds:us-east-1:123456789012:db:mysql-instance-2018-10-06-12-24"],

  "detail": {

    "EventCategories": ["creation"],

    "SourceType": "SNAPSHOT",

    "SourceArn": "arn:aws:rds:us-east-1:123456789012:db:mysql-instance-2018-10-06-12-24",

    "Date": "2018-10-06T12:26:13.882Z",

    "SourceIdentifier": "rds:mysql-instance-2018-10-06-12-24",

    "Message": "Automated snapshot created"

  }

}

Lambda Function

The lambda function is a python code that gets the events and extracts the useful information and starts a copy in another region.

The lambda functions just start the copy, the function doesn't wait to be completed the task.

You can see the python code here

Enabling logs and alerting in AWS EKS cluster - CloudWatch Log Insights and Metric filters

Daniel German Rivera — Tue, 10 Jan 2023 00:04:53 +0000

In the first part of this post, I described the steps to enable logs in an EKS cluster, control plane logs, and container logs using Fluent-bit and CloudWatch, in this post I will show how to get helpful information from logs and create alerts for specific events.

AWS CloudWatch Logs could be used to store logs generated by resources created in AWS or external resources, once the logs are in CloudWatch you can run some queries to get specific information and create alerts for specific events.

CloudWatch Log Insights

CloudWatch Logs Insights allows search and analysis of log data stored in Amazon CloudWatch Logs, queries can be run to identify potential causes and validate fixes, an advantage of Logs Insights is the ability to discover fields, doing more easy the process to run queries. Automatically Logs Insights define 5 fields:

message: This field contains the original log message sent to CloudWatch.
timestamp: contains the event timestamp registered in the original event.
ingestionTime: contains the time when CloudWatch Logs received the log event.
logStream: contains the name of the log stream where the event was added.
log: it is an identifier in the form of account-id:log-group-name. When querying multiple log groups, this can be useful to identify which log group a particular event belongs to.

Those fields are discovered by CloudWatch and depending on the log type that we are using CloudWatch will discover more fields, for instance, for EKS control plane logs you can see the field shown in the following image:

Running queries in AWS Log Groups

Queries can be run to search specific events, the field discovery is really helpful in designing the query to run, in some cases when you don't know the structure of the logs you can run a simple query to get the fields that CloudWatch discovered, and use them to design the query based on the use case.

An important thing to mention here is if the CloudWatch log groups have been encrypted by KMS you must have permission to use the key.

How to run queries?

Go to AWS CloudWatch service, in the left panel select Logs Insights.
Select the logs groups to run queries, up to 20 log groups can be selected, Logs Insights will search in the groups specified.
By default CloudWatch shows a simple query, you can run it and validate the fields discovered by CloudWatch. The following image shows a query that gets up to 10 results, you can check it and validate the fields.

AWS documentation describes the query sintax that you can use.

Queries examples for EKS

Search API calls made by kubectl user-agent

The following example searches the calls made to Kube API using the kubectl command and with the GET action. In this case, the log group is the one that EKS has created when you enable logging in the cluster, in the previous post I mentioned the name-format and how to enable it.

fields @logStream, @timestamp, @message
| filter @logStream like /kube-apiserver-audit/
| filter userAgent like /kubectl/
| sort @timestamp desc
| filter verb like /(get)/

The first line is used to specify the fields that you want to show in the results, the query in the example will show the logStream name, the timestamp, and the message, you can add the fields that you want.

Search events filtering by namespace

You can use the fields discovered by CloudWatch to create your queries, in this case for EKS control logs, one field discovered is objectRef.namespace, and the following query uses it to get the events where the kube-system namespace is used.

fields @timestamp, @message
| sort @timestamp desc
| filter objectRef.namespace like 'kube-system'
| limit 2

The result of the previous query could look like:

Creating alerts for specific events

CloudWatch logs can look for specific patterns inside the events that are sent, this allows the creation of metrics to monitor if a particular event happened and create alerts for this, for that we need to use AWS CloudWatch metric filters, this is configured directly in the Log group created and you must specify a pattern.

To create a metric filter you must select the Log Group to use, then in actions, you will see the option to create a metric filter.

Defining a pattern for the filter

When you are creating the filter you need to define a pattern to specify what to look for in the log file. When the logs are in JSON format is more easily define the pattern because you just need to specify the name of the key that you want to evaluate, for this case you can use the following format:

{ PropertySelector Operator Value }

For more details about the pattern syntax you can check the AWS documentation.

When the logs are not in JSON format is more tricky define the pattern , in this case you need to take in mind that each space is taken as a word in the filter, for instance, suppose that you have the following log event:

time="2022-10-09T20:37:25Z" level=info msg="STS response" accesskeyid=ABCD1234 accountid=123456789 arn="arn:aws:sts::123456789:assumed-role/test" client="127.0.0.1:1234" method=POST path=/authenticate

In this case, you have different words separated by space, if you want to look for some word specific you need to know the exact position of the element to compare, let's see this with an example, in this case, i want to match the logs with a level equal to info, if you see the previous log event you can validate that leve=info is the word number 2 in the whole event, in this case, the pattern could be:

[word1,word2="level=info",word3]

Remember that you need to include the whole word that you want to compare in this case you can use leve=info or you put the word between * which means any match with the word specified. let's see the result of the previous pattern.

If you see, CloudWatch is showing each word defined in the pattern and the events that match.

Let's see more examples to be more clear

Metric Filter to alert when actions are made in AWS-AUTH configmap

AWS-AUTH configmap is used to authenticate the user by IAM RBAC, and part of this kind of event looks like the following message:

kind:Event
level: Metadata
objectRef.apiVersion: v1
objectRef.name: aws-auth
objectRef.namespace: kube-system
objectRef.resource: configmaps
verb: get

Unauthorized modifications in this configmap could be a security risk, a metric filter can be created to alert when this configmap is edited. The pattern could be:

{( $.objectRef.name = "aws-auth" && $.objectRef.resource = "configmaps" ) && ($.verb = "delete" || $.verb = "create" || $.verb = "patch" ) }

Metric filter for 403 code response in calls to K8 API-SERVER

This is useful to detect several attempts to login or make calls to the cluster without valid credentials, part of this event looks like the following message:

requestURI: /
responseStatus.code:403
responseStatus.reason: Forbidden
responseStatus.status: Failure
sourceIPs.0 : 12.34.56.78
verb: get

The pattern could be:

{$.responseStatus.code = "403" }

Metric filter to check Access Denied generated by AWS IAM-RBAC

You can monitor the number access-denied in API calls, this is generated by the AWS IAM-RBAC, part of this event looks like the following message:

authenticator-8c7,2022-08-08 10:04:56,"time=""2020-08-04T28:43:44Z"" level=warning msg=""access denied"" client=""127.0.0.1:1234"" error=""sts getCallerIdentity failed: error from AWS (expected 200, got 403)"" method=POST path=/authenticate"

The pattern could be:

[time,level="*warning*",message1="*access*",message2="*denied*",more]

Enabling logs and alerting in AWS EKS cluster - CloudWatch and fluent-bit

Daniel German Rivera — Thu, 10 Nov 2022 14:35:36 +0000

Logs are a fundamental component in our environments, these provide useful information that helps to debug in the case of any issue or to identify events that can affect the security of the application. Logs should be enabled in each component, from the infrastructure level to the application level. This brings some challenges like where to store the logs, what kind of events log, how to search events, and what to do with the logs.

In this post I will share my experience enabling and configuring logging in an EKS cluster, creating alerts to send a notification when a specific event appears in the logs.

This is the part #1 in which I show how to enable control plane logging and container logging in an EKS cluster, in the part #2 I will show you how to enable some alerts using the logs groups created.

Before to start is important to mention that logs can contain private data like user information, keys, passwords, etc. For this reason, logs should be encrypted at rest and enable restrictions to access them.

Kubernetes Control Plane logging

Kubernetes architecture can be divided into a control plane and worker nodes, control plane contains the components that manage the cluster, components like etcd, API Server, Scheduler, and Controller Manager. Almost every action done in the cluster pass through the API Server that logs each event.

AWS EKS manages the control plane for us, deploying and operating the necessary components. By default, EKS doesn't have logging enabled and actions from our side are required. Enabling EKS control plane logging is an easy task, you need to know what component log and enable it. You can enable logs for the API server, audit, authenticator, control manager, and scheduler.

In my opinion, audit logs and authenticator are useful because records actions done in our cluster and help us to understand the origin of the actions and requests generated by the IAM authenticator.

By terraform you can use the following code to create a simple cluster and enabling audit,api,authenticator, and scheduler logs.


 Terraform
resource "aws_eks_cluster" "kube_cluster" {
  name                      = "test-cluster"
  role_arn                  = aws_iam_role.role-eks.arn
  version                   = "1.22"
  enabled_cluster_log_types = ["audit", "api", "authenticator","scheduler"]
  vpc_config {
    subnet_ids              = ["sub-1234","sub-5678"]
    endpoint_private_access = true
    endpoint_public_access  = true
  }
}

Logs are stored in AWS CloudWatch logs and the log group is created automatically following this name structure /aws/eks/<cluster-name>/cluster, inside the group you can find the log stream for each component that you enabled



authenticator-123abcd
kube-apiserver-123abcd
kube-apiserver-123abcd

By default, the Log group created by AWS doesn't have encryption and retention days enabled, I recommend creating the logs group by yourself and specifying and KMS Key, and setting some time to expiry the logs, Kubernetes generates a considerable number of logs that will increase the size of the group that can impact the billing.

Kubernetes Containers logging

The steps mentioned above were to enable just logging in the control plane, to send logs generated by the applications running in the containers a log aggregator is necessary, in this case, I will use Fluent-Bit](https://fluentbit.io/how-it-works/)

Fluent-Bit runs as a daemonSet in the cluster and sends logs to CloudWatch Logs. Fluent-Bit creates the log groups using the configuration specified in the kubernetes manifests.

Here is important to mention that AWS has created a docker image for the daemonSet, this can be found in this link.

AWS describes the steps to run the daemonSet, this is done by some commands, but I will use a Kubernetes manifest that can be stored in our repository and then use Argo or Fluxcd to automate deployments.

The following steps show the manifests to create the objects that Kubernetes needs to send containers logs to CloudWatch, you must have access to the cluster and by kubeclt command create the resources (kubeckt apply -f manifest-name.yml).

1. Namespace creation

A K8 namespace is necessary, amazon-cloudwatch name will use for this, you can change the name but make sure to use the same in the following steps.



apiVersion: v1
kind: Namespace
metadata:
  name: amazon-cloudwatch
  labels:
    name: amazon-cloudwatch

2. ConfigMap for aws-fluent-bit general configs

This configMap is necessary to specify some configurations for fluent-bit and for AWS, for instance, the cluster-name AWS use to create the logs group. In this case, I don't want to create an HTTP server for fluent-bit and I will read the logs from the tail, more information about this can be found }(https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file).



apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-general-configs ## you can use a different name, make sure to use the same in the following steps
  namespace: amazon-cloudwatch
data:
  cluster.name: ${CLUSTERNAME}
  http.port: ""
  http.server: "Off"
  logs.region: ${AWS_REGION}
  read.head: "Off"
  read.tail: "On"

3.Service Account, Cluster Role and Role Binding

Some permissions are required to send logs from daemonSet to Cloudwatch, you can attach a role to the worker-nodes or use a service account with an IAM role, in this case, I will create an IAM role and associate it with a service account.
The following Terraform code creates a policy and the role.


 Terraform
resource "aws_iam_role" "iam-role-fluent-bit" {
  name                  = "role-fluent-bit-test"
  force_detach_policies = true
  max_session_duration  = 3600
  path                  = "/"
  assume_role_policy    = jsonencode({

{
    Version= "2012-10-17"
    Statement= [
      {
        Effect= "Allow"
        Principal= {
            Federated= "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/oidc.eks.${REGION}.amazonaws.com/id/${EKS_OIDCID}"
        }
        Action= "sts:AssumeRoleWithWebIdentity"
        Condition= {
          StringEquals= {
            "oidc.eks.${REGION}.amazonaws.com/id/${EKS_OIDCID}:aud": "sts.amazonaws.com",
"oidc.eks.${REGION}.amazonaws.com/id/${EKS_OIDCID}:sub": "system:serviceaccount:${AWS_CLOUDWATCH_NAMESPACE}:${EKS-SERVICE_ACCOUNT-NAME}"
          }
        }
      }
    ]
  }

})

}

EKS_OIDCID: is the OpenID Connect for your cluster, you can get it in the cluster information or by terraform outputs.
AWS_CLOUDWATCH_NAMESPACE: is the namespace create in the step 1, in this case amazon-cloudwatch.
ACCOUNT_ID: is the AWS account number where the cluster was created.

The role needs a policy with permissions to create and put logs in cloudwatch, you can use the following code to create the policy and attach it to IAM Role created.


 Terraform

resource "aws_iam_policy" "policy_sa_logs" {
  name        = "policy-sa-fluent-bit-logs"
  path        = "/"
  description = "policy for EKS Service Account fluent-bit "
  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "ec2:DescribeVolumes",
                "ec2:DescribeTags",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams",
                "logs:DescribeLogGroups",
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:PutRetentionPolicy"
            ],
            "Resource": "arn:aws:logs:${REGION}:${ACCOUNT_ID}:*:*"
        }
    ]
}
EOF
}

######## Policy attachment to IAM role ########

resource "aws_iam_role_policy_attachment" "policy-attach" {
  role       = aws_iam_role.iam-role-fluent-bit.name
  policy_arn = aws_iam_policy.policy_sa_logs.arn
}

Once the role has been created the Service account can be created, you can use the following k8 manifest for that, you should replace the IAM_ROLE variable for the ARN of the role created previously.



apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: amazon-cloudwatch
  annotations:
    eks.amazonaws.com/role-arn: "${IAM_ROLE}"

With the SA ready, you need to create a cluster role and associate that to the SA created, the following manifests can be used for that.



apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-role
rules:
  - nonResourceURLs:
      - /metrics
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
      - pods/logs
      - nodes
      - nodes/proxy
    verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-role
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: amazon-cloudwatch

4.ConfigMap for fluent-bit configurations

A ConfigMap is used to specify a detailed configuration for Fluent-bit, AWS already defines a configuration, but you can add custom configs, The following link, shows the configurations defined by AWS, if you see, the first objects created in the YAML are the manifest defined in previous steps, in this step you just need to define the ConfigMap with name fluent-bit-config, I don't want to put here all the manifest because is a little long and can complicate the lecture of this post.

With this ConfigMap, Fluent Bit will create the log groups in the below table, you also have the option to create by terraform and specify encryption and retention period (i recommend this way).

CloudWatch Log Group Name	Source of the logs(Path inside the Container)
aws/containerinsights/Cluster_Name/application	All log files in /var/log/containers
/aws/containerinsights/Cluster_Name/host	Logs from /var/log/dmesg, /var/log/secure, and /var/log/messages
/aws/containerinsights/Cluster_Name/dataplane	The logs in /var/log/journal for kubelet.service, kubeproxy.service, and docker.service.

If you analyze the ConfigMap you can see the INPUTS for each source mentioned in the table.



apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: amazon-cloudwatch
  labels:
    k8s-app: fluent-bit

... OTHER CONFIGS 

### here is the INPUT configurations for application logs  
application-log.conf: |
    [INPUT]
        Name                tail
        Tag                 application.*
        Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
        Path                /var/log/containers/*.log

... OTHER CONFIGS 

    [OUTPUT]
        Name                cloudwatch_logs
        Match               application.*
        region              $${AWS_REGION}
        log_group_name      /aws/containerinsights/$${CLUSTER_NAME}/application
        log_stream_prefix   $${HOST_NAME}-
        auto_create_group   false
        extra_user_agent    container-insights
        log_retention_days  ${logs_retention_period}

The OUTPUT in the previous manifest defines the CloudWatch log Group configuration that fluent bit will create, as you can see you can specify if the log groups should be created, the prefix for the stream, the name, and the retention period for the logs. If you are using Terraform you should set to false the option auto_create_group

5.DaemonSet Creation

This is the last step :) , AWS also provides the manifest to create the DaemonSet, in this link you can find it in the bottom of the file. As I mentioned, I don't want to put the whole file here, you can copy and paste the content or edit the file if you have custom configurations.



apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: amazon-cloudwatch
  labels:
    k8s-app: fluent-bit
... OTHER CONFIGS

Once you have run the above steps, you can validate that the daemonSet is running well and if everything is ok you should be the Logs groups in the AWS console with some events passed by fluent-bit DaemonSet

References

Trusting in your IaC -Terraform-Compliance

Daniel German Rivera — Sun, 30 Jan 2022 21:50:37 +0000

Infrastructure as Code has started to be an important part of our cloud journey, giving us the ability to automate deploys and replicate our infra in multiple environments and accounts. Using code to define our infra allow us to implement some practices from developers world, like testing, version control, etc.

As a lot of tools, IaC can introduce bugs or maybe miss configurations that then can be our headache when errors arise, or maybe audit companies come to review the security of our infra.
IaC has enabled us to implement development practices like generating Pull Request with the new changes, this is very important when we want the revision from other coworkers, frameworks to do testing and maybe IaC tools that use common language programing help us to validate our infra-code, but this is more for technical teams and translate this to non-technical languages can be complicated, limiting the scope for the revision. Here is when terraform-compliance framework appears to help us to define rules to
validate our infra and define if accomplish with the requirements defined from the technical side and business side.

Terraform-compliance is a framework that validates the compliance in our infrastructure defined by terraform, this is based on negative testing and BDD that work together to validate our infra. This post will show how to implement this framework and add it to our DevOps pipeline used to deploy our IaC, AWS will be used as a cloud provider.

Requirements

Terraform-compliance validates the code before it is deployed, we can install it using docker or via pip. In this post, we will use pip.

Python
terraform 0.12+

How does this work?

Terraform-compliance uses policies that define the features that the infrastructure must-have, like encryption enabled in the buckets, resources well tagged, etc. These policies are executed against the plan that Terraform generates and are defined using Behaviour Driven Development(BDD) Principles, that use English like the language to define the policies.

To work with terraform we need to create a file in which the policies are defined, using BDD principles that file is named feature and contain the scenario that we want to evaluate.
The structure of the file is the following:

Feature: A summary about the things to validate
Scenario/scenario Outline: define the test to execute, this includes BDD directives like:
GIVEN: It is used to define a context that can be a list of resources or data that we want to check, we can see this as a filter.
WHEN: filter the context defined above, for instance, if the context defined says that will evaluate all the s3 buckets, with WHEN we can filter by tags for instance. If the condition doesn't pass this just skip to the next line instead of fail.
THEN: Has a similar function that WHEN but if the condition doesn't pass the scenario will fail.
AND: It is used to define an extra condition to our scenario, this is an optional statement.

Here and example to evaluate if all the resources has tag defined.

Scenario: Ensure all resources have tags Given I have resource that supports tags defined Then it must contain tags And its value must not be null

Each BDD directive has more capabilities and can be checked in Terraform-compliance documentation.

The Feature file is executed against terraform plan, to do this terraform-compliance needs the TF plan as an input, for that we need to save our plan in an external file and pass it to terraform-compliance command that we will see later.

With the basic explanation made above let's make and example.

Hands on !!!

This post was part of a webinar made in my current company and a Github repository was created, here is the link.This repo contains a few AWS resources defined by terraform and uses Github actions to execute with each push the terraform-compliance framework. All the stuff that we will execute is before the terraform apply so we don't spend money creating resources :)

The project has been divided into two parts, the first one contains the terraform code to deploy the infrastructure and is located in the root of the repo, the second part is a folder named compliance that contains some files with .feature extension and defines the rules that we want to evaluate against our TF plan.

Terraform Code

#### Data Sources

data "aws_caller_identity" "ID_CURRENT_ACCOUNT" {}


###  KMS Policy 
data "aws_iam_policy_document" "kms_policy" {
  statement {
    sid    = "Enable IAM User Permissions"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.ID_CURRENT_ACCOUNT.account_id}:root"]
    }
    actions = [
      "*"
    ]
    resources = ["*"]

  }
}


##########################
## Secret manager

# KMS ky to encrypt at rest secret manager
module "kms_secret_manager" {
  source = "./modules/kms"
  NAME   = "KMS-SecretManager-${var.environment}"
  POLICY = data.aws_iam_policy_document.kms_policy.json
}

module "secret_manager" {
  source    = "./modules/secret_manager"
  NAME      = "secret_${var.environment}"
  RETENTION = 10
  KMS_KEY   = module.kms_secret_manager.ARN_KMS
}


module "secret_manager_k8" {
  source    = "./modules/secret_manager"
  NAME      = "secret_k8_${var.environment}"
  RETENTION = 10
  KMS_KEY   = module.kms_secret_manager.ARN_KMS
}

resource "aws_s3_bucket" "bucket_test" {
  bucket = "my-bucket-forcompliance-test"

  server_side_encryption_configuration {
    rule {
      /*   
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
      */

      apply_server_side_encryption_by_default {
        kms_master_key_id = module.kms_secret_manager.ARN_KMS
        sse_algorithm     = "aws:kms"
      }


    }

  }

  tags = {
    Name        = "bucket_${var.environment}"
    Environment = "develop"
    Owner       = "DanielR"
  }
}

resource "aws_s3_bucket" "bucket_test2" {
  bucket = "my-bucket-forcompliance-test"

  server_side_encryption_configuration {
    rule {

      apply_server_side_encryption_by_default {
        kms_master_key_id = module.kms_secret_manager.ARN_KMS
        sse_algorithm     = "aws:kms"
      }

    }

  }

  tags = {
    Name        = "bucket_${var.environment}"
    Environment = "develop"
    Owner       = "DanielR"
  }
}


resource "aws_secretsmanager_secret" "secret_manager2" {
  name                    = "test"
  recovery_window_in_days = 10
  lifecycle {
    create_before_destroy = true
  }
  tags = {
    Name        = "test"
    Environment = "develop"
    Owner       = "DanielR"
  }
}

Terraform code creates KMS key, Secret managers that are encrypted by KMS and s3 buckets with SSE with KMS.

Compliance folder has three files and each one define a scenario, let's move on with the S3.feature file.

Feature:  This validates if the s3 buckets has Encryption enabled using KMS
Scenario: Ensure that s3 buckets are encrypted by KMS
    Given I have aws_s3_bucket resource configured
    When it contain server_side_encryption_configuration
    Then it must have apply_server_side_encryption_by_default
    Then it must have sse_algorithm
    And its value must be "aws:kms"

With that file TF-Compliance framework will validate if the S3 buckets created with that TF code have Server-Side-Encryption enabled but using KMS key, here a quick explanation of the file:

Given statement define the context in this case, all the resources defined by aws_s3_bucket
When statement makes more small the context, in this case TF-compliance will check the buckets that have server_side_encryption_configuration in their definition.
We have two Then statement and are used to defined that the buckets must to have apply_server_side_encryption_by_default and the sse_algorithm must be aws:kms.

Running the commands

As we mentioned before, we are using Github actions to run the TF-compliance command, the idea is to validate automatically if the rules are accomplished before the plan if so, the pipe will allow the TF apply command and the infra will comply with the rules defined.

To do that, we need to generate the terraform plan, this must be stored in a file in the same directory, then that will be an input for our TF-compliance command.

To do that we can run:

Terraform init

terraform plan -out=plan.out

plan.out is the name of the file to create, we can put any another name.

Once the plan is created we can go ahead with the execution of the TF-Compliance command, here is the command to do the magic

terraform-compliance -f compliance/ -p plan.out

With the -f flag we specify the folder in which we are storing the rules(.features files).

The output will be

Let's change the code to use the default algorithm to encrypt the s3 buckets.

resource "aws_s3_bucket" "bucket_test" {
  bucket = "my-bucket-forcompliance-test"

  server_side_encryption_configuration {
    rule {

      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }

    }

  }

}

Running again the TF-compliance command will be:

With this we can validate how the framework works, we can define multiple rules and these can be checked for all the teams because are not 100% technical.