DEV Community: Joaquin Menchaca

Architecting Kubernetes Deployments with Python

Joaquin Menchaca — Sun, 12 Jul 2026 21:56:20 +0000

Python is an excellent language for automating cloud infrastructure, but the official Kubernetes Python client leaves developers with an important architectural decision:

Where should Kubernetes manifests live?

Should they be constructed directly with Python objects? Embedded as multiline strings? Or stored as external files and rendered at runtime?

Each approach works, but they have very different implications for readability, maintainability, and long-term operational cost.

The key is recognizing that deployment logic and platform configuration evolve on different lifecycles. Your deployment code, the part that authenticates to Kubernetes, renders templates, and applies resources, may remain unchanged for months. Your manifests, however, often change weekly as applications evolve, resource limits are tuned, cloud-provider annotations are added, or networking requirements change.

When those two concerns are tightly coupled, even a configuration, only change forces you to modify, test, and redeploy the delivery or application code itself. Over time, this increases maintenance costs, slows platform changes, and makes configuration drift and production mistakes more likely.

This is a familiar software engineering principle: separate concerns that evolve independently. The same thinking that keeps application configuration separate from executable code also applies to Kubernetes manifests. Treating manifests as first-class configuration artifacts allows them to evolve independently from the Python code that delivers them.

In this article we'll compare three ways of deploying Kubernetes resources with the official kubernetes-python-client, ranging from tightly coupled implementations to a design that cleanly separates deployment logic from platform configuration.

The Landscape at a Glance

The comparison below assumes a common application deployment scenario, where the desired state is largely known ahead of time. Controllers and Operators have fundamentally different requirements where the desiredstate is determined dynamically at runtime.

Method¹	Maintainability²	Readability³	Flexibility⁴	Best Used For
Direct API Object Calls	⭐⭐⭐ Medium	⭐ Low	⭐⭐⭐⭐⭐ High	Dynamic, complex operators, custom controllers
Embedded Strings	⭐⭐ Low	⭐⭐⭐ Medium	⭐⭐ Low	Small utilities, remediation scripts, one-off automation
External YAML Templates (Jinja2)	⭐⭐⭐⭐⭐ High	⭐⭐⭐⭐⭐ High	⭐⭐⭐ Medium	Declarative, Standardized application deployments, CI/CD pipelines

Method refers to the overall approach used to define and deploy Kubernetes resources, not just features of the official kubernetes-python-client.
Maintainability measures the cost to safely modify the solution over time.
Readability measures ability of the engineer quickly determine the desired state and confidently modify it?
Flexibility measures the ability to determine or modify the desired state dynamically at runtime.

1. The Most Verbose: Direct API Object Calls

The official client libary (kubernetes-python-client) exposes the Kubernetes API as native Python objects. This provides complete programmatic control over every resource, but that flexibility comes at the cost of verbose, deeply nested object construction.

Instead of expressing the desired state directly as a Kubernetes manifest, it is represented through a hierarchy of Python objects that are ultimately translated back into the Kubernetes API.

This additional abstraction makes the desired state less obvious during code reviews and collaboration, since engineers must mentally reconstruct the resulting Kubernetes resource instead of reading it directly as a manifest.

The example below shows simple web application deployment and a service (type: LoadBalancer) with some annotations for AWS NLB.

from kubernetes import client, config

config.load_kube_config()
apps_v1 = client.AppsV1Api()
core_v1 = client.CoreV1Api()

# Define the Deployment Object
deployment = client.V1Deployment(
    metadata=client.V1ObjectMeta(name="demo-nlb-app-deployment"),
    spec=client.V1DeploymentSpec(
        replicas=3,
        selector=client.V1LabelSelector(match_labels={"app": "demo-nlb-app"}),
        template=client.V1PodTemplateSpec(
            metadata=client.V1ObjectMeta(labels={"app": "demo-nlb-app"}),
            spec=client.V1PodSpec(
                containers=[client.V1Container(name="web", image="nginx:alpine")]
            )
        )
    )
)

# Define the Service Object with AWS NLB Annotations
service = client.V1Service(
    metadata=client.V1ObjectMeta(
        name="demo-nlb-app-service",
        annotations={
            "service.beta.kubernetes.io/aws-load-balancer-type": "external",
            "service.beta.kubernetes.io/aws-load-balancer-scheme": "internet-facing",
            "service.beta.kubernetes.io/aws-load-balancer-nlb-target-type": "instance"
        }
    ),
    spec=client.V1ServiceSpec(
        type="LoadBalancer",
        selector={"app": "demo-nlb-app"},
        ports=[client.V1ServicePort(port=80, target_port=80)]
    )
)

# Apply to Cluster
apps_v1.create_namespaced_deployment(namespace="default", body=deployment)
core_v1.create_namespaced_service(namespace="default", body=service)

Notice that the desired state is no longer represented as a Kubernetes manifest. Instead, it is distributed across a hierarchy of Python object constructors. Understanding the resulting Deployment requires understanding both the Kubernetes resource model and the Python API simultaneously. This makes the deployed infrastructure less transparent during reviews, since the desired state must first be reconstructed from the implementation.

The Verdict: This approach excels when the desired state cannot be known until runtime, such as Kubernetes Controllers, Operators, or other applications that continuously reconcile resources based on external events. For conventional application deployments, however, the additional flexibility rarely outweighs the increased verbosity and maintenance cost compared to declarative manifests.

Segue: Community Client Libraries

As the official client is generated directly from the Kubernetes OpenAPI specification, its object model closely mirrors the Kubernetes API rather than idiomatic Python. This has led the community to develop alternative client libraries that provide a more Pythonic developer experience while still interacting with the same Kubernetes APIs.

pykube-ng - A lightweight client that models Kubernetes resources much like a Python ORM such as SQLAlchemy or Django's ORM. Resources are manipulated through familiar CRUD-style operations rather than constructing deeply nested API objects.
kr8s - Lightweight client whose API closely mirrors the kubectl experience while abstracting away much of the connection setup and boilerplate required by the official client. It also includes native asyncio support for asynchronous applications.

2. The Hardcoded Copy-Paste: Embedded Strings

The easiest shortcut is copying raw YAML directly into a Python file using triple-quoted """

As the manifest remains recognizable YAML, this approach initially feels like the best of both worlds: the readability of Kubernetes manifests combined with the convenience of keeping everything in one Python file. However, that convenience comes from embedding platform configuration directly inside the application or delivery logic, creating a coupling that becomes increasingly expensive as the application evolves.

import yaml
from kubernetes import client, config

config.load_kube_config()

# Hardcoded variables injected via Python f-strings
app_name = "demo-nlb-app"
image_name = "nginx:alpine"
replicas = 3

manifest_string = f"""
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {app_name}-deployment
spec:
  replicas: {replicas}
  selector:
    matchLabels:
      app: {app_name}
  template:
    metadata:
      labels:
        app: {app_name}
    spec:
      containers:
      - name: web
        image: {image_name}
---
apiVersion: v1
kind: Service
metadata:
  name: {app_name}-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
spec:
  type: LoadBalancer
  selector:
    app: {app_name}
  ports:
  - port: 80
    targetPort: 80
"""

# Split the string and apply each document sequentially
apps_v1 = client.AppsV1Api()
core_v1 = client.CoreV1Api()

for doc in yaml.safe_load_all(manifest_string):
    if doc['kind'] == 'Deployment':
        apps_v1.create_namespaced_deployment(namespace="default", body=doc)
    elif doc['kind'] == 'Service':
        core_v1.create_namespaced_service(namespace="default", body=doc)

The Verdict: Embedded manifests are a pragmatic choice for small utilities, remediation scripts, or one-off automation where the manifest is simply an implementation detail. For long-lived deployment tooling, however, embedding configuration inside the application or delivery logic tightly couples two concerns that often evolve independently, increasing maintenance costs over time.

Segue: The Cost of Tethering Configuration to Delivery Logic

Embedding manifests directly into Python couples two concerns that frequently evolve at different rates: the deployment logic and the platform configuration. The deployment engine may remain unchanged for months, while manifests evolve regularly as applications grow, cloud-provider annotations change, or operational requirements shift.

Asymmetric Velocity: Deployment logic and platform configuration rarely evolve together. A deployment script may remain unchanged for months while Kubernetes manifests are updated frequently to adjust resource limits, networking policies, or cloud-provider annotations. Coupling them forces both to follow the same release lifecycle.
The Testing Tax: If your configuration is baked inside the application code, modifying a simple JSON block or YAML string often requires rebuilding, retesting, and redeploying the full application, which can slow your release down and impact time-to-market and revenue. If testing is bypass to run a fix out, your blast radius explodes.
The Silent Outage: Finding configuration buried inside hundreds of lines of code logic increases cognitive load. In practice, this exact pattern frequently leads to real-world outages because a developer updates an application specification but forgets to dig through the delivery codebase to update the embedded string matching it.

For a simple atomic web application deployment, the desired state should be immediately visible during a code review. Reviewers should not have to mentally search through delivery logic to understand what Kubernetes will ultimately receive. Keeping manifests as first-class artifacts preserves that transparency.

3. The Gold Standard: External YAML Templates (Jinja2)

When the desired state is largely known ahead of time, the cleanest approach is to keep Kubernetes manifests as first-class artifacts and render only the values that must be determined dynamically. Standard .yaml files remain in your repository—preserving IDE support, schema validation, and the familiar Kubernetes representation, while Jinja2 provides lightweight parameter substitution at runtime.

This allows you to link your service and deployment fluidly using shared variables like {{ app_name }}.

`manifest.yaml.j2` (The Clean Template):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ app_name }}-deployment
spec:
  replicas: {{ replicas }}
  selector:
    matchLabels:
      app: {{ app_name }}
  template:
    metadata:
      labels:
        app: {{ app_name }}
    spec:
      containers:
      - name: web
        image: {{ image_name }}
---
apiVersion: v1
kind: Service
metadata:
  name: {{ app_name }}-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
spec:
  type: LoadBalancer
  selector:
    app: {{ app_name }}
  ports:
  - port: 80
    targetPort: 80

`deploy.py` (The Clean Script)

import yaml
from jinja2 import Template
from kubernetes import client, config

config.load_kube_config()

# 1. Read the external Jinja2 template file
with open("manifest.yaml.j2", "r") as f:
    raw_template = f.read()

# 2. Render variables into the template seamlessly
rendered_yaml = Template(raw_template).render(
    app_name="demo-nlb-app",
    replicas=3,
    image_name="nginx:alpine"
)

# 3. Parse and safely push to Kubernetes
apps_v1 = client.AppsV1Api()
core_v1 = client.CoreV1Api()

for doc in yaml.safe_load_all(rendered_yaml):
    if doc['kind'] == 'Deployment':
        apps_v1.create_namespaced_deployment(namespace="default", body=doc)
    elif doc['kind'] == 'Service':
        core_v1.create_namespaced_service(namespace="default", body=doc)

Notice that the deployment logic and the desired state are now represented independently. The Python code is responsible only for supplying runtime values and applying resources, while the manifest remains a transparent description of what Kubernetes will receive. Each artifact can evolve independently without forcing changes to the other.

The Verdict: For conventional application deployments, this approach provides the best balance of maintainability, readability, and flexibility. The deployment implementation remains focused on supplying runtime values, while the manifests continue to express the desired state as first-class configuration artifacts. This separation reduces the cost of change, improves collaboration across engineering teams, and leaves the door open to adopting tools such as Helm or GitOps workflows in the future.

Beyond the Official Client

The approaches discussed in this article intentionally stay close to the official Kubernetes Python client (kubernetes-python-client). As your applications grow, higher-level frameworks such as Kopf, cdk8s, and Pulumi provide different programming models that may better fit specific classes of problems.

Kopf — A Python framework for building Kubernetes Operators and Controllers. Rather than manually watching resources and reconciling state, Kopf provides an event-driven framework designed specifically for applications whose desired state is determined at runtime.
cdk8s — Generates Kubernetes manifests from object-oriented Python code. This allows infrastructure to be modeled using software abstractions while still producing standard Kubernetes YAML as the final artifact.
Pulumi — Treats infrastructure as software with state management, dependency tracking, and cloud resource provisioning. Unlike cdk8s, Pulumi manages infrastructure lifecycle rather than simply generating Kubernetes manifests.

Conclusion

When automating Kubernetes deployments with Python, the goal isn't to eliminate YAML or force every deployment through an object-oriented API. The goal is to choose a representation that matches the nature of the problem.

If the desired state is largely known ahead of time, keep it declarative. External manifests preserve Kubernetes as the source of truth while allowing the Python implementation to focus on supplying runtime values and interacting with the Kubernetes API. Because the deployment logic and platform configuration evolve independently, treating manifests as first-class artifacts reduces the cost of change, improves transparency during code reviews, and encourages collaboration across engineering disciplines.

When the desired state cannot be known until runtime, however, the trade-offs change. Controllers, Operators, and other event-driven applications continuously reconcile resources based on external events. In these cases, the flexibility of the Kubernetes Python API justifies the additional complexity because the desired state must be synthesized rather than declared.

Ultimately, there isn't a single "best" approach. Each representation optimizes for a different class of problems. Understanding those trade-offs, and choosing the abstraction that matches the problem—is far more important than the particular library or templating engine you use.

Practical Guidance

Use external manifest templates when the desired state is largely known ahead of time and only a subset of values must be determined dynamically.
Use embedded manifests for small utilities, remediation scripts, or one-off operational automation where the manifest is simply an implementation detail.
Use the Kubernetes Python API directly when resources must be created or modified dynamically in response to runtime events, such as Controllers or Operators.

How AI Bridged a Tooling Gap: Extending InSpec for Kubernetes CRDs

Joaquin Menchaca — Mon, 06 Jul 2026 21:31:31 +0000

The Problem: Trailing Ecosystems in Infrastructure-as-Code

As infrastructure and platform engineers, we spend a lot of time dealing with libraries that are just feature-complete enough to be useful, but still have rough edges.

Recently, I was writing compliance tests using InSpec to validate some Kubernetes infrastructure. While InSpec provides solid compliance-as-code capabilities, its Kubernetes resource pack hasn't evolved much in recent years. If you are working with modern cloud-native architectures, like testing the newer Gateway API standards, you quickly realize it's missing foundational building blocks.

In my case, I needed to validate Custom Resource Definitions (CRDs).

The Mirage: A Confident Hallucination

When I first paired with an AI model to spin up the compliance tests, it spit out exactly what I wanted to see. The loop looked clean, intuitive, and natural:

expected_standard_crds.each do |crd_name|
  describe k8s_custom_resource_definition(name: crd_name) do
    it { should exist }
    its('spec.group') { should cmp 'gateway.networking.k8s.io' }
  end
end

There was just one issue: k8s_custom_resource_definition does not exist in the InSpec Kubernetes resource pack. The model had completely hallucinated the API wrapper.

Normally, this is the point where you sigh and prepare to write a messy workaround. In InSpec, that usually means dropping into raw Ruby, shelling out to kubectl via command resources, or parsing raw JSON strings, completely destroying the clean, expressive nature of compliance-as-code.

The Twist: Engineering the Abstraction with Claude

When I brought this problem to Claude, it didn't just apologize and rewrite the test using a messy workaround. Instead, it spotted the framework's gap and pivoted to an engineering mindset.

It realized that to make that ideal test code work, it just needed to build the missing abstraction.

By dropping into InSpec's underlying K8s client backend, Claude generated a custom resource implementation and placed it right into the profile's libraries/ directory:

# libraries/k8s_custom_resource_definition.rb

class K8sCustomResourceDefinition < Inspec.resource(1)
  name "k8s_custom_resource_definition"
  desc "Verifies a Kubernetes CustomResourceDefinition exists."

  example "
    describe k8s_custom_resource_definition(name: 'gateways.gateway.networking.k8s.io') do
      it { should exist }
      its('group') { should cmp 'gateway.networking.k8s.io' }
    end
  "

  def initialize(opts = {})
    @crd_name = opts.fetch(:name)

    begin
      @crd = inspec.backend.client.api("apiextensions.k8s.io/v1")
        .resource("customresourcedefinitions")
        .get(@crd_name)
    rescue ::K8s::Error::NotFound
      @crd = nil
    end
  end

  def exists?
    !@crd.nil?
  end

  def group
    @crd&.spec&.group
  end

  def resource_id
    @crd_name
  end

  def to_s
    "Kubernetes CustomResourceDefinition #{@crd_name}"
  end
end

Because Claude correctly mapped the apiextensions.k8s.io/v1 resource endpoint, the original, elegant test loop now worked flawlessly.

The output of a successful run:

🔄 Fetching ephemeral session details from active 'aws login' context...
📍 Target context resolved: Region=us-east-2, Cluster=mycluster
🚀 Launching stage 2 (bindings-ready) compliance sweep via cinc-auditor...

Profile:   Stage 2: AWS LBC Preparation Succeeded (bindings-ready-profile)
Version:   0.1.0
Target:    kubernetes://0123456789ABCDEF0123456789ABCDEF.gr7.us-east-2.eks.amazonaws.com:443
Target ID: 188e8b62-f664-5221-82ad-9eec791facd6

  ✔  gateway-api-standard-crds: Verify Kubernetes Gateway API v1.5.0 Standard CRDs
     ✔  backendtlspolicies.gateway.networking.k8s.io
     ✔  gatewayclasses.gateway.networking.k8s.io
     ✔  gateways.gateway.networking.k8s.io
     ✔  grpcroutes.gateway.networking.k8s.io
     ✔  httproutes.gateway.networking.k8s.io
     ✔  listenersets.gateway.networking.k8s.io
     ✔  referencegrants.gateway.networking.k8s.io
     ✔  tlsroutes.gateway.networking.k8s.io
  ✔  gateway-api-experimental-crds: Verify Kubernetes Gateway API v1.5.0 Experimental CRDs
     ✔  backendtlspolicies.gateway.networking.k8s.io
     ✔  gatewayclasses.gateway.networking.k8s.io
     ✔  gateways.gateway.networking.k8s.io
     ✔  grpcroutes.gateway.networking.k8s.io
     ✔  httproutes.gateway.networking.k8s.io
     ✔  listenersets.gateway.networking.k8s.io
     ✔  referencegrants.gateway.networking.k8s.io
     ✔  tcproutes.gateway.networking.k8s.io
     ✔  tlsroutes.gateway.networking.k8s.io
     ✔  udproutes.gateway.networking.k8s.io
     ✔  xbackendtrafficpolicies.gateway.networking.x-k8s.io
     ✔  xmeshes.gateway.networking.x-k8s.io
  ✔  aws-lbc-gateway-extensions-crds: Verify AWS Load Balancer Controller Gateway API CRDs
     ✔  listenerruleconfigurations.gateway.k8s.aws
     ✔  loadbalancerconfigurations.gateway.k8s.aws
     ✔  targetgroupconfigurations.gateway.k8s.aws
  ✔  lbc-iam-binding-ready: ServiceAccount is bound to an IAM role via IRSA or Pod Identity
     ✔  kube-system/aws-load-balancer-controller is expected to exist
     ✔  IRSA role-arn annotation is expected not to be empty
  ✔  lbc-iam-policy-attached: The controller's IAM role actually grants the permissions AWS LBC needs
     ✔  IAM role discovered from the ServiceAccount binding is expected not to be empty
     ✔  AWS IAM Role eksctl-mycluster-addon-iamserviceaccount-kub-Role1-iI9ixQ19jzJ8 is expected to exist
     ✔  IAM policies attached to role eksctl-mycluster-addon-iamserviceaccount-kub-Role1-iI9ixQ19jzJ8 is expected not to be empty
     ✔  iam:CreateServiceLinkedRole  →  *
     ✔  ec2:DescribeAccountAttributes  (+28 more)  →  *
     ✔  cognito-idp:DescribeUserPoolClient  (+16 more)  →  *
     ✔  ec2:AuthorizeSecurityGroupIngress  (+1 more)  →  *
     ✔  ec2:CreateSecurityGroup  →  *
     ✔  ec2:CreateTags  →  security-group/*
     ✔  ec2:CreateTags  (+1 more)  →  security-group/*
     ✔  ec2:AuthorizeSecurityGroupIngress  (+2 more)  →  *
     ✔  elasticloadbalancing:CreateLoadBalancer  (+1 more)  →  *
     ✔  elasticloadbalancing:CreateListener  (+3 more)  →  *
     ✔  elasticloadbalancing:AddTags  (+1 more)  →  [3 resources]
     ✔  elasticloadbalancing:AddTags  (+1 more)  →  [4 resources]
     ✔  elasticloadbalancing:ModifyLoadBalancerAttributes  (+10 more)  →  *
     ✔  elasticloadbalancing:AddTags  →  [3 resources]
     ✔  elasticloadbalancing:RegisterTargets  (+1 more)  →  targetgroup/*/*
     ✔  elasticloadbalancing:SetWebAcl  (+5 more)  →  *
     ✔  elasticloadbalancing:DescribeListenerAttributes  (+1 more)  →  *


Profile:   Amazon Web Services Resource Pack (inspec-aws)
Version:   1.83.63
Target:    kubernetes://0123456789ABCDEF0123456789ABCDEF.gr7.us-east-2.eks.amazonaws.com:443
Target ID: 188e8b62-f664-5221-82ad-9eec791facd6

     No tests executed.

Profile:   InSpec Profile (inspec-k8s)
Version:   0.1.3
Target:    kubernetes://0123456789ABCDEF0123456789ABCDEF.gr7.us-east-2.eks.amazonaws.com:443
Target ID: 188e8b62-f664-5221-82ad-9eec791facd6

     No tests executed.

Profile Summary: 5 successful controls, 0 control failures, 0 controls skipped
Test Summary: 45 successful, 0 failures, 0 skipped

The Real Leverage of AI in Platform Engineering

We often talk about AI in the context of writing boilerplate application code or scaffolding quick utility scripts. But for infrastructure engineers wrestling with fragmenting or trailing open-source ecosystems, the real leverage might be ecosystem expansion.

When open-source tools are 80% complete, we usually inherit the technical debt of bridging the final 20% ourselves. If LLMs can shift from "writing code within the strict constraints of a tool" to "dynamically rewriting and extending the tool itself," it completely changes how we maintain our platform tooling stack.

Have you experienced a similar workflow where one model's hallucination became the blueprint for another model to extend a framework?

Let's discuss in the comments!

Minimalist EKS: The Easy Way

Joaquin Menchaca — Fri, 26 Jun 2026 19:56:54 +0000

Amazon EKS manages the Kubernetes control plane, but you remain responsible for provisioning the underlying AWS infrastructure, including networking, security groups, IAM roles, and worker nodes.

The eksctl command-line tool automates this entire process. Using a clean, declarative syntax, it handles the complex multi-step orchestration across different AWS services that would normally require extensive manual configuration.

In this guide, we will use eksctl to deploy both the VPC and EKS layers simultaneously, establishing a complete testing sandbox with a single command.

About eksctl

“eksctl is a command-line utility tool that automates and simplifies the process of creating, managing, and operating Amazon Elastic Kubernetes Service (Amazon EKS) clusters. Written in Go, eksctl provides a declarative syntax through YAML configurations and CLI commands to handle complex EKS cluster operations that would otherwise require multiple manual steps across different AWS services.” (source What is Eksctl?)

Tools

These are the tools used in this article.

AWS CLI [aws]: Interact with AWS services.
kubectl [kubectl]: Manage Kubernetes clusters and workloads.
eksctl [eksctl]: Provision Amazon EKS clusters and supporting infrastructure.

Optional:

helm [helm]: Install and manage Kubernetes applications.

Verifying Tool Versions

This guide has been tested with these tools:

aws: 2.35.7
kubectl: v1.36.2
helm: v4.2.0
eksctl: 0.227.0

You can verify your versions using a POSIX Shell, such as bash or zsh:

printf "aws: %s\nkubectl: %s\nhelm: %s\neksctl: %s\n" \
  "$(aws --version 2>&1 | awk -F'[/ ]' '{print $2}')" \
  "$(kubectl version --client | awk '/Client Version:/ {print $3}')" \
  "$(helm version --short | cut -d+ -f1)" \
  "$(eksctl version | cut -d- -f1)"

Prerequisite: AWS Profile

Before running the deployment script, you need an AWS CLI profile configured for an IAM identity with sufficient administrative permissions to provision EKS, IAM roles, and VPC infrastructure. NEVER use your AWS root account for local development.

📓 NOTE: Recent versions of the AWS CLI (v2.32.0+) support aws login, which uses your AWS Management Console credentials to obtain temporary credentials. This eliminates the need to store long-lived static access keys on your machine.. (see Sign in through the AWS Command Line Interface)

1. Configure the Profile Setting

Run the following below to create a config profile block dynamically:

# Define your local variables
MY_ACCOUNT_ID="123456789012" # Replace with your real AWS Account ID
MY_REGION="us-east-2"

mkdir -p ~/.aws

# Append the login profile structure to your AWS config file
cat <<EOF >> ~/.aws/config
[profile myuser]
login_session = arn:aws:iam::$ACCOUNT_ID:user/myuser
region = $MY_REGION
EOF

2. Authenticate and Activate

Trigger the browser-based authentication handshake:

aws login --profile myuser

A browser window will open automatically, prompting you to complete your standard console sign-in. Once authorized, activate the profile in your active shell:

export AWS_PROFILE=myuser

3. Verify Your Active Session

Confirm that your terminal is actively communicating with AWS using your temporary token:

aws sts get-caller-identity

Upon successful connection, the AWS CLI will output your active identity details:

{
    "UserId": "AIDA0123456789EXAMPLE",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/myuser"
}

If the command returns your metadata, your authentication profile is active and ready to spin up your testing sandbox.

The Minimalist One-Command EKS Cluster

If you want the absolute fastest path to an operational Kubernetes sandbox, you can rely on the native CLI defaults of eksctl.

The following snippet isolates your configuration file, provisions a default public/private network topology, and configures the control plane using a single baseline execution:

This is the least amount of steps to create a Kubernetes cluster.

# Set your active profile and environment targets
export AWS_PROFILE="myuser"
export MY_CLUSTER_NAME="mycluster"
export MY_REGION="$(aws configure get region)"

# Direct your Kubernetes context to an isolated configuration file
mkdir -p $HOME/.kube/aws/
export KUBECONFIG="$HOME/.kube/aws/$MY_REGION.$MY_CLUSTER_NAME.yaml"

# Spin up the infrastructure stack
eksctl create cluster \
  --name $MY_CLUSTER_NAME \
  --region $MY_REGION \
  --version 1.36

🚨 IMPORTANT SECURITY NOTE: While this simple deployment is perfect for rapid testing, it relies on legacy permissions mapping. By default, it attaches the broad AWS VPC CNI network policy directly to the underlying worker node’s shared IAM Instance Profile role. This means any standard application pod running on the cluster technically inherits those raw EC2 network management capabilities at the hardware layer.

Once the creation process finishes, logging into your AWS Management Console will reveal your newly active cluster topology, complete with healthy control plane paths and managed compute node resources:

To verify the core system components running on your cluster, query all namespaces:

kubectl get all --all-namespaces

Running this command displays the baseline networking, metrics, and DNS workloads initialized on the cluster:

Deploying a Demo Nginx Application

With your cluster up and running, you can verify external traffic routing by deploying a simple Nginx application and exposing it via an AWS Classic Load Balancer (ELBv1):

# Create the Nginx deployment
kubectl create deployment demo-lb-app --image=nginx:alpine

# Expose the deployment as a Kubernetes LoadBalancer Service
kubectl expose deployment demo-lb-app --port=80 --type=LoadBalancer

To find the public AWS DNS name assigned to your load balancer, check the status of your services:

kubectl get services

This will display the external hostname provisioning in your AWS account:

Copy the address from the EXTERNAL-IP field and paste it into a web browser to verify connectivity:

Alternatively, you can query the endpoint directly from your terminal using curl:

# Capture the hostname dynamically
DEMO_ADDR=$(kubectl get service demo-lb-app \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
)

# Send an HTTP request to the load balancer
curl -i $DEMO_ADDR

The command returns the raw HTTP response headers and HTML body from the container:

Addendum: Free-Tier EKS Cluster

If you are strictly working within the AWS Free Tier, you can provision a minimal single-node cluster using t3.micro instances.

📓 NOTE: The t3.micro instances have severe memory and IP assignment limitations. To make a single micro node usable without running out of pod capacity instantly, you should pass the --zones flag to restrict network placement to two availability zones instead of three (saving NAT gateway overhead costs), and explicitly drop the node count to one:

# Set your environment variables
export AWS_PROFILE="myuser"
export MY_CLUSTER_NAME="mycluster"
export MY_REGION="$(aws configure get region)"

# Isolate your Kubernetes configuration file
mkdir -p $HOME/.kube/aws/
export KUBECONFIG="$HOME/.kube/aws/$MY_REGION.$MY_CLUSTER_NAME.yaml"

# Create a minimal, single-node free tier cluster
eksctl create cluster \
  --name $MY_CLUSTER_NAME \
  --region $MY_REGION \
  --zones "${MY_REGION}a,${MY_REGION}b" \
  --nodes 1 \
  --node-type "t3.micro"

⚠️ WARNING: This configuration is strictly for basic CLI testing and experimentation. Because a t3.micro only supports a maximum of 4 pods per network interface, default system components (like coredns, kube-proxy, and vpc-cni) will consume almost the entire available capacity, leaving minimal room to run custom applications.

Addendum: Robust Minimalist Cluster

For development or short-lived playground clusters, running standard eksctl create cluster command line defaults is perfectly adequate. However, for formal test and staging environments, you will want an infrastructure profile that is more robust, secure, and production-like:

Strict Isolation via Pod Identity: Only the explicit vpc-cni driver is permitted to modify network configurations, locking out malicious or unprivileged application pods from tampering with AWS routing infrastructure.
High Availability Architecture: Three managed worker nodes are automatically calculated and balanced across three distinct Availability Zones (AZs) to prevent single-datacenter outages from dropping your services.
Native Persistent Storage: The AWS EBS CSI driver is packaged directly into the cluster lifecycle to easily back stateful application requirements.
Modern Cost Optimization: Upgrades compute selections to the modern m6i.large general-purpose instance family (saving on price-per-performance relative to the older m5.large defaults).

Instead of managing an endless string of CLI arguments, you can capture these configurations cleanly inside a version-controlled file. Below is a one-touch command that dynamically interpolates your local environment variables into a ClusterConfig layout and pipes it directly into the eksctl builder:

export AWS_PROFILE="myuser"
export MY_CLUSTER_NAME="mycluster"
export MY_REGION="$(aws configure get region)"
export MY_EKS_VERSION="1.36"

eksctl create cluster --config-file /dev/stdin <<EOF
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: $MY_CLUSTER_NAME
  region: $MY_REGION
  version: $MY_EKS_VERSION

managedNodeGroups:
  - name: ng-1
    instanceType: m6i.large
    desiredCapacity: 3
    minSize: 3
    maxSize: 3
    labels:
      alpha.eksctl.io/cluster-name: $MY_CLUSTER_NAME
      alpha.eksctl.io/nodegroup-name: ng-1
    tags:
      alpha.eksctl.io/nodegroup-name: ng-1
      alpha.eksctl.io/nodegroup-type: managed

iam:
  withOIDC: true

addonsConfig:
  autoApplyPodIdentityAssociations: true

addons:
  - name: vpc-cni
    useDefaultPodIdentityAssociations: true
  - name: aws-ebs-csi-driver
    useDefaultPodIdentityAssociations: true
  - name: eks-pod-identity-agent
EOF

Addendum: Optional Enhancements

The default minimalist cluster is sufficient for basic experimentation. If you choose not to use the automated ClusterConfig heredoc template from the earlier addendum, you can manually install these individual components via the CLI to support real-world application workloads:

1. IAM OIDC Provider

If your specific testing tools require legacy IAM Roles for Service Accounts (IRSA) rather than the modern EKS Pod Identity agent, you must manually associate an OpenID Connect (OIDC) identity provider with your cluster:

eksctl utils associate-iam-oidc-provider \
  --cluster $MY_CLUSTER_NAME \
  --region $MY_REGION \
  --approve

2. Pod Identity Agent Addon

The Pod Identity Agent enables Kubernetes service accounts to access AWS resources using EKS Pod Identity.

eksctl create addon \
  --cluster $MY_CLUSTER_NAME \
  --region $MY_REGION \
  --name eks-pod-identity-agent

3. Amazon EBS CSI Driver Addon

The EBS CSI Driver allows your cluster to dynamically provision and manage lifecycle storage volumes backed by Amazon EBS.

eksctl create addon \
  --cluster $MY_CLUSTER_NAME \
  --region $MY_REGION \
  --name aws-ebs-csi-driver

Cleanup

To avoid recurring AWS charges after you finish experimenting with your sandbox, delete all provisioned resources using the following command:

eksctl delete cluster --name $MY_CLUSTER_NAME --region $MY_REGION

This single command automatically tears down both the EKS compute resources and the underlying VPC network stack.

Essential DevOps Tools for Ubuntu

Joaquin Menchaca — Wed, 14 Jan 2026 20:25:54 +0000

Popular tools for provisioning the cloud are Terraform and Kubernetes tools kubectl and helm. For Linux, online instructions for how to install these tools can be inconsistent and potentially error prone. This article shows how to install these tools on Debian and Ubuntu based distros, which by extension would include Mint, Pop! OS, Zorin OS and others.

This guide walk you through how to install the latest versions of popular DevOps tool on Ubuntu Linux. This was tested on Ubuntu 24.04 Noble Nambat.

Segue: Why Linux?

For developers accustomed to macOS or Windows, choosing Linux as a primary development platform may seem unusual. For others, Linux is the default choice. With Windows now supporting Linux through WSL, Linux-based development environments have become increasingly common.

Beyond personal preference, the strongest case for Linux is environment consistency. Production systems, and especially CI/CD pipelines, almost always run on Linux. Developing and testing directly in a Linux environment allows you to validate build, test, and deployment scripts under the same conditions they'll run in production.

This consistency shortens development cycles by reducing surprises late in the pipeline and eliminates platform-specific "gotchas" that exist on macOS or Windows but never appear in Linux-based CI/CD or production systems.

The Tools

This is an overview of the tools.

Cloud Provisioning
- Terraform [terraform]: tool for provisioning cloud infrastructure.
- AWS CLI [aws]: tool to interact with Amazon Web Services
- okta-aws-cli [okta-aws-cli]: allows you to get temporary IAM credentials using Okta as the Identity provider.
- Azure CLI Tools [az]: tool to interact with Azure
- Google Cloud CLI Tools [gcloud]: tool to interact with GCP
Containers
- docker [docker]: tools to build container images and run containers
Kubernetes
- Kubernetes CLI [kubectl]: command line tool for interacting with Kubernetes control plan
- Helm [helm]: Tool for configuring, installing, and distributing Kubernetes applications.
- Kustomize [kustomize]: tool to configure Kubernetes application through patching.
- Helmfile [helmfile]: is a declarative spec for deploying helm charts that allows you to template helm chart values as well integrate patching support
- Krew [kubectl krew]: is a plugin system for Kubernetes
Java
- Correto OpenJDK [java, javac, jar, keytool]: Corretto is an OpenJDK distribution that is supported on many platforms.
Others
- Vault [vault]: Allow securing, storing, and controlling access to secrets artifacts: tokens, passwords, certificates, and encryption keys.

The Installation

Below are script steps that you can use to install the packages and tools. These scripts follow the Google Shell Style Guide and I took some measures to reduce the width so that it would be nicely in this guide.

Repositories

Many of these tools are available from private repositories. You can add them with the following script below:

#!/usr/bin/env bash

main() {
  sudo apt update

  # Hashicorp
  # Uses Ubuntu codename dynamically
  UBUNTU_CODENAME=$(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release)

  # terraform, vault
  # https://developer.hashicorp.com/vault/install#linux
  # https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli
  add_apt_repo "hashicorp" \
               "https://apt.releases.hashicorp.com/gpg" \
               "https://apt.releases.hashicorp.com" \
               "$UBUNTU_CODENAME main"

  # kubectl
  # https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management
  add_apt_repo "kubernetes" \
               "https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key" \
               "https://pkgs.k8s.io/core:/stable:/v1.35/deb/" \
               "/"

  # helm
  # https://helm.sh/docs/intro/install/#from-apt-debianubuntu
  add_apt_repo "helm" \
               "https://packages.buildkite.com/helm-linux/helm-debian/gpgkey" \
               "https://packages.buildkite.com/helm-linux/helm-debian/any/" \
               "any main"

  # Corretto
  # https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/generic-linux-install.html
  add_apt_repo "corretto" \
               "https://apt.corretto.aws/corretto.key" \
               "https://apt.corretto.aws" \
               "stable main"

  # google-cloud-cli
  # https://docs.cloud.google.com/sdk/docs/install-sdk#deb
  add_apt_repo "googlecloud" \
               "https://packages.cloud.google.com/apt/doc/apt-key.gpg" \
               "https://packages.cloud.google.com/apt" \
               "cloud-sdk main"

  # azure-cli
  # https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?view=azure-cli-latest&pivots=apt
  add_apt_repo "azure" \
               "https://packages.microsoft.com/keys/microsoft.asc" \
               "https://packages.microsoft.com/repos/azure-cli/" \
               "$UBUNTU_CODENAME main"

  # docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  # Docs: https://docs.docker.com/engine/install/ubuntu/
  add_apt_repo "docker" \
               "https://download.docker.com/linux/ubuntu/gpg" \
               "https://download.docker.com/linux/ubuntu" \
               "$UBUNTU_CODENAME stable"
}

add_apt_repo() {
  local NAME=$1
  local KEY_URL=$2 # Signed-By
  local REPO_URI=$3
  local DISTRO_STRING=$4 # "Suites Components" or "/"

  local KEYRING="/usr/share/keyrings/${NAME}-archive-keyring.gpg"
  local LIST_FILE="/etc/apt/sources.list.d/${NAME}.list"
  local ARCH=$(dpkg --print-architecture)

  echo "Configuring repository: $NAME"

  # Download and dearmor the GPG key
  wget -qO - "$KEY_URL" | sudo gpg --dearmor --yes -o "$KEYRING"

  # Construct and save the list file
  # Note: arch=$ARCH is included by default 
  #       as it is standard for most 3rd party repos
  echo "deb [arch=$ARCH signed-by=$KEYRING] $REPO_URI $DISTRO_STRING" \
    | sudo tee "$LIST_FILE" > /dev/null

  sudo apt-get update \
    -o Dir::Etc::sourcelist="$LIST_FILE" \
    -o Dir::Etc::sourceparts="-" \
    -o APT::Get::List-Cleanup="0"
}


main

Tools Installed from Repositories

For the tools that can be installed from these repositories, we can run through the following script below.

#!/usr/bin/env bash

sudo apt install \
 apt-transport-https \
 ca-certificates \
 curl \
 gnupg \
 lsb-release \
 jq \
 software-properties-common \
 unzip

# Install Tools from Private Repos
sudo apt install terraform
sudo apt install vault
sudo apt install kubectl
sudo apt install helm
sudo apt install java-17-amazon-corretto-jdk
sudo apt install google-cloud-cli
sudo apt install azure-cli
sudo apt install docker-ce \
                 docker-ce-cli \
                 containerd.io \
                 docker-buildx-plugin \
                 docker-compose-plugin

Tools Installed as Binaries

#!/usr/bin/env bash

main() {
  # install AWS CLI
  install_binary_aws_cli
  install_binary_okta_aws_cli

  # install other binaries
  install_binary_kustomize
  install_binary_helmfile
}

install_binary_kustomize() {
  URL_PATH="kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"
  URL_SCRIPT="https://raw.githubusercontent.com/$URL_PATH"
  curl -s "$URL_SCRIPT" | bash
  sudo mv ~/kustomize /usr/local/bin
}

install_binary_helmfile() {
  ARCH="amd64"
  VER="1.2.3"
  PKG="helmfile_${VER}_linux_$ARCH.tar.gz"
  URL_PATH="helmfile/helmfile/releases/download/v$VER/$PKG"
  URL="https://github.com/$URL_PATH"
  TEMP_DIR=$(mktemp -d)
  wget -qO- "$URL" | tar -xz -C "$TEMP_DIR"
  sudo mv $TEMP_DIR/helmfile /usr/local/bin
  rm -rf "$TEMP_DIR"
}

install_binary_okta_aws_cli() {
  ARCH="amd64"
  VER="2.5.1"
  PKG="okta-aws-cli_${VER}_linux_$ARCH.tar.gz"
  URL_PATH="okta/okta-aws-cli/releases/download/v$VER/$PKG"
  URL="https://github.com/$URL_PATH"
  TEMP_DIR=$(mktemp -d)
  wget -qO- "$URL" | tar -xz -C "$TEMP_DIR"
  sudo mv $TEMP_DIR/okta-aws-cli /usr/local/bin
  rm -rf "$TEMP_DIR"
}

install_binary_aws_cli() {
  TEMP_DIR=$(mktemp -d)
  PKG="awscli-exe-linux-x86_64.zip"
  wget "https://awscli.amazonaws.com/$PKG" -P $TEMP_DIR
  unzip $TEMP_DIR/$PKG -d $TEMP_DIR
  sudo $TEMP_DIR/aws/install
  rm -rf TEMP_DIR
}

main

Verify Installations

# verify installations
terraform --version
vault --version
kubectl version --client
helm version
kustomize version
helmfile --version
java --version
aws --version
okta-aws-cli --version

The Plugins

Both helm and Kubernetes can extend their functionality through a plugin. Here’s how you can get some popular plugins for these tools.

Helm Plugins

Helm has a few plugins that you can install, as shown below. I highly recommend the helm-diff, as this allows you to see what resources you will modify before installing or upgrading the the application.

Here are some examples:

helm plugin install https://github.com/databus23/helm-diff
helm plugin install https://github.com/aslafy-z/helm-git
helm plugin install https://github.com/hypnoglow/helm-s3.git
helm plugin install https://github.com/jkroepke/helm-secrets

Kubernetes Plugin Engine

Kubernetes have a plugin system with Kubernetes Krew, which you can install using the script below.

#!/usr/bin/env bash

main() {
  install_krew

  # Add this in your startup scripts 
  export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
}

install_krew() {
  TEMP_DIR=$(mktemp -d)
  pushd $TEMP_DIR

  OS="$(uname | tr '[:upper:]' '[:lower:]')"
  ARCH="$(
    uname -m \
    | sed -e 's/x86_64/amd64/' \
          -e 's/\(arm\)\(64\)\?.*/\1\2/' \
          -e 's/aarch64$/arm64/'
  )"
  KREW="krew-${OS}_${ARCH}"
  PKG="/${KREW}.tar.gz"
  URL_PATH="kubernetes-sigs/krew/releases/latest/download/$PKG"
  URL="https://github.com/$URL_PATH"
  curl -fsSLO "$URL"
  tar zxvf "${KREW}.tar.gz"
  ./"${KREW}" install krew

  popd
  rm -rf TEMP_DIR
}

main

Kubernetes Plugins

Once krew is installed, you can install some popular plugins.

Here are some examples:

kubectl krew install neat
kubectl krew install sort-manifests
kubectl krew install ktop
kubectl krew install tree
kubectl krew install graph
kubectl krew install cert-manager 
kubectl krew install get-all
kubectl krew install ice
kubectl krew install deprecations
kubectl krew install ns
kubectl krew install access-matrix
kubectl krew install pod-lens
kubectl krew install resource-capacity

Conclusion

Ubuntu is a powerful platform for development, but it can be surprisingly complex for new developers, especially when it comes to installing and maintaining modern DevOps tooling. Most online instructions are fragmented, outdated, or inconsistent in how they handle repositories, keys, and binary installs. This can lead inconsistencies across developer workstations as well as production environments.

So my goal here is to reduce the complexity by providing consistent set of installation steps that follow best practices and rely on official sources wherever possible.

This can help avoid environment drift, especially when install scripts like this pin the tool versions and are routinely tested. This will help avoid those situations where you’ll hear: “works on my laptop”…

Essential DevOps Tools for macOS

Joaquin Menchaca — Sat, 10 Jan 2026 00:43:09 +0000

This guide walks through installing popular tools for cloud provisioning and container management on macOS. While many may not realize it, macOS traces its roots back to NeXTSTEP and is built on the Mach-derived XNU kernel. The Darwin userland provides a BSD-based Unix environment with familiar POSIX tools, along with macOS-specific additions such as the launchd init and service management system. This Unix heritage is what allows many Linux-focused cloud and container tools to run reliably on macOS with minimal differences.

The challenge arises when scripts written for Linux-based cloud environments are run on macOS. Because macOS ships with BSD-flavored utilities, such as grep, sed, and awk, scripts that work correctly on Linux may behave differently or fail entirely. To address this, this guide not only covers installing cloud provisioning tools like Terraform and Kubernetes tooling, but also walks through installing a common set of command-line tools so that scripts behave consistently across both Linux and macOS.

The Tools

This is an overview of the tools.

Cloud Provisioning

Terraform [terraform]: tool for provisioning cloud infrastructure.
AWS CLI [aws]: tool to interact with Amazon Web Services
okta-aws-cli [okta-aws-cli]: allows you to get temporary IAM credentials using Okta as the Identity provider.

Kubernetes

Kubernetes CLI [kubectl]: command line tool for interacting with Kubernetes control plan
Helm [helm]: Tool for configuring, installing, and distributing Kubernetes applications.
[Kustomize] [kustomize]: tool to configure Kubernetes application through patching.
Helmfile [helmfile]: is a declarative spec for deploying helm charts that allows you to template helm chart values as well integrate patching support
Krew [kubectl krew]: is a plugin system for Kubernetes

Java

Beyond compiling languages and running java programs, there are tools to manage certificate, keytool, and a useful tool for managing tar archives: jar.

Correto OpenJDK [java, javac, jar, keytool]: Corretto is an OpenJDK distribution that is supported on many platforms.

Others

Vault [vault]: Allow securing, storing, and controlling access to secrets artifacts: tokens, passwords, certificates, and encryption keys.

The Package System

You can install Homebrew with the following commands:

# Install XCode command line tools
xcode-select --install

# Install Homebrew
URL="https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh"
/bin/bash -c "$(curl -fsSL $URL)"

The Common Core Commands

These commands represent a common set of tools I use across Linux and Windows MSYS2 environments. They include GNU utilities such as bash, find, gawk, grep, and sed, along with commonly used tools like bc, curl, git, jq, and zsh. While macOS includes some of these commands by default, they are often BSD variants with different behavior or significantly older versions that lack features commonly relied upon in scripts.

# Install common core commands (GNU + bc, curl, jq, zsh) 
brew install \
  autoconf \
  bash \
  bc \
  binutils \
  coreutils \
  curl \
  diffutils \
  ed \
  findutils \
  flex \
  gawk \
  git \
  gnu-indent \
  gnu-sed \
  gnu-tar \
  gnu-which \
  gpatch \
  grep \
  gzip \
  jq \
  less \
  m4 \
  make \
  nano \
  screen \
  watch \
  wdiff \
  wget \
  zip \
  zsh

The Available Shells

You can run this to make these updated shells available to your environment by appending them to /etc/shells.

echo "$(brew --prefix)/bin/bash" | sudo tee -a /etc/shells
echo "$(brew --prefix)/bin/zsh" | sudo tee -a /etc/shells

The Default Shell

While I prefer Zsh as my default interactive shell due to its rich prompt and plugin ecosystem provided by Oh My Zsh!, all of my scripts are written and tested using GNU Bash for portability and consistency.

You can change your default shell using one of the commands below to the shell of your choice:

chsh -s $(brew --prefix)/bin/bash
chsh -s $(brew --prefix)/bin/zsh

When running shell scripts, use env command for your shebang to select the shell from your path, rather than override with a dinosaur version of the shell provided by the system.

Examples:

Bash shebang line: #!/usr/bin/env bash
Zsh shebang line: #!/usr/bin/env zsh

The Paths

These new GNU commands and other Keg-only commands will not be available for scripts, so they need to be added to the PATH.

You will want to add this below to your startup scripts, like .bash_profile or .zshrc for macOS.

update_homebrew_path() {
  local BREW_PREFIX
  local GNU_PATHS
  local KEG_PATHS
  local COMBINED_PATHS
  local DEDUPE_SCRIPT

  if ! command -v brew &>/dev/null; then return fi

  BREW_PREFIX="$(brew --prefix)"
  GNU_PATHS=$(printf "%s:" "${BREW_PREFIX}/opt/"*"/libexec/gnubin")
  KEG_PATHS=$(
    brew info --installed --json=v1 \
      | jq -r 'map(select(.keg_only == true)) | .[].name' \
      | while read -r PKG; do
          if [[ -d "${BREW_PREFIX}/opt/${PKG}/bin" ]]; then
            printf "%s:" "${BREW_PREFIX}/opt/${PKG}/bin"
          fi
        done
  )

  COMBINED_PATHS="${GNU_PATHS}${KEG_PATHS}"
  DEDUPE_SCRIPT='print join ":", grep { ! $seen{$_}++ } split /:/, shift'

  perl -e "${DEDUPE_SCRIPT}" "${COMBINED_PATHS}" | sed 's/^://; s/:$//'
}

NEW_PATHS="$(update_homebrew_path)"
export PATH="${NEW_PATHS}:${PATH}"

The Installation

Now that we have the basic consistent core commands available, we can install cloud provisioning (terraform), Kubernetes tools (kubectl, helm, helmfile, kustomize), Corretto (java, jar, keytool), and other tools (vault, jq).

# install AWS CLI
brew install awscli
brew install okta-aws-cli

# install packages
brew tap hashicorp/tap
brew install hashicorp/tap/terraform
brew install hashicorp/tap/vault
brew install kubernetes-cli  # kubectl
brew install helm # helm

# Java
brew tap homebrew/cask-versions
brew install --cask corretto17

# Helmfile and Kustomize
brew install kustomize
brew install helmfile

# Other
brew install jq

Verification

You can verify the tool installations with the following.

# verify installations
terraform --version
vault --version
kubectl version --client
helm version
kustomize version
helmfile --version
java --version
aws --version
okta-aws-cli --version

Plugins

Both helm and Kubernetes can extend their functionality through a plugin. Here’s how you can get some popular plugins for these tools.

Helm Plugins

helm plugin install https://github.com/databus23/helm-diff
helm plugin install https://github.com/aslafy-z/helm-git
helm plugin install https://github.com/hypnoglow/helm-s3.git
helm plugin install https://github.com/jkroepke/helm-secrets

Kubernetes Plugin Engine

Kubernetes has a plugin system with Kubernetes Krew. You can use this script below to install Krew.

#!/usr/bin/env bash

main() {
  install_krew
  # Add this in your startup scripts 
  export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
}

install_krew() {
  TEMP_DIR=$(mktemp -d)
  pushd $TEMP_DIR
  OS="$(uname | tr '[:upper:]' '[:lower:]')"
  ARCH="$(
    uname -m \
    | sed -e 's/x86_64/amd64/' \
          -e 's/\(arm\)\(64\)\?.*/\1\2/' \
          -e 's/aarch64$/arm64/'
  )"
  KREW="krew-${OS}_${ARCH}"
  PKG="/${KREW}.tar.gz"
  URL_PATH="kubernetes-sigs/krew/releases/latest/download/$PKG"
  URL="https://github.com/$URL_PATH"
  curl -fsSLO "$URL"
  tar zxvf "${KREW}.tar.gz"
  ./"${KREW}" install krew
  popd
  rm -rf TEMP_DIR
}

main

Extra: The Package Manifest

Homebrew supports using a package manifest that list all the packages in a single file called Brewfile. Create a file named Brewfile with the following contents:

# Common Core Commands
brew 'autoconf'
brew 'bash'
brew 'bc'
brew 'binutils'
brew 'coreutils'
brew 'curl'
brew 'diffutils'
brew 'ed'
brew 'findutils'
brew 'flex'
brew 'gawk'
brew 'git'
brew 'gnu-indent'
brew 'gnu-sed'
brew 'gnu-tar'
brew 'gnu-which'
brew 'gpatch'
brew 'grep'
brew 'gzip'
brew 'jq'
brew 'less'
brew 'm4'
brew 'make'
brew 'nano'
brew 'screen'
brew 'watch'
brew 'wdiff'
brew 'wget'
brew 'zip'
brew 'zsh'

# AWS Tools
brew 'awscli'
brew 'okta-aws-cli'
# Hashicorp
tap 'hashicorp/tap'
brew 'hashicorp/tap/terraform'
brew 'hashicorp/tap/vault'
# Kubernetes
brew 'kubernetes-cli'
brew 'helm'
brew 'kustomize'
brew 'helmfile'
# Java
tap 'homebrew/cask-versions'
cask 'corretto17'
# Other
brew 'jq'

When ready, you can install packages listed in the Brewfile with the following command:

brew bundle --verbose

Remember to set up the PATH correctly to include the paths to GNU and Keg-only commands. Also, update the available shells in /etc/shells to that your new version of zsh and bash are available.

Loki: Getting Started

Joaquin Menchaca — Sun, 06 Aug 2023 23:22:44 +0000

The realm of open source log aggregators has seen significant growth over the years, with key players like ElasticSearch (2010) and Graylog (2016) dominating the field. More recently, Grafana Loki (2019) has emerged as a compelling addition to this ecosystem.

This article aims to facilitate a rapid onboarding process for log shipping using Promtail, log aggregation with Loki, and data visualization through Grafana. Collectively, this powerful trio of tools is commonly referred to as the PLG (Promtail-Loki-Grafana) stack.

The Server System

These instructions will work on Debian or Ubuntu system.

Using Vagrant (Virtualbox)

If you have Intel system, you can use Vagrant with Virtualbox to quickly bring up virtual servers on your workstation. Follow the instructions from respect sites to install these tools.

When ready, create a file called Vagrantfile with the following contents:



Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu2204"
  config.vm.network "forwarded_port", guest: 3000, host: 3000, host_ip: "127.0.0.1"
  config.vm.network "forwarded_port", guest: 3100, host: 3100, host_ip: "127.0.0.1"
end

NOTE: For this setup, the configuration will utilize 3000 for Grafana and 3100 for Loki on the localhost. If any services are currently running on these ports, you'll have to either halt those services or modify the host: key in the provided above configuration to use an alternative port.

When ready, you can bring up the system and log into the system with the following:



vagrant up
vagrant ssh

Installing Loki

On your desired system, such as the Vagrant managed virtual machine above, you can run the following to install PLG:



sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key \
  https://apt.grafana.com/gpg.key

# add a repository for stable releases
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" \
  | sudo tee -a /etc/apt/sources.list.d/grafana.list

sudo apt-get update
sudo apt-get install -y promtail loki grafana

Verify Services are running



sudo service loki status
sudo service grafana-server status

If any of these services are stopped, you can run:



sudo service loki start
sudo service grafana-server start

The Client Workstation

Visualization

You can access the login page by opening a web browser on your system and navigating to http://127.0.0.1:3000. Simply use the default credentials, admin for both the username and password. Upon login, you will be prompted to set a new password for the admin account.

After logging in, you can click on Explore, Loki, and then select on of the logs by clicking on Label.

Ship More Logs with Promtail

When setting up a new system, there may not be many interesting logs initially. However, your host workstation system is likely to have accumulated a wealth of interesting logs over time.

If you are running the server as a virtual machine using the previously mentioned Vagrantfile configuration, you can easily ship logs from the laptop to localhost:3100, where Loki is up and running. This allows you to leverage the abundant logs from your host workstation for analysis and visualization.

You can install Promtail locally as well. If the host system is running Debian or Ubuntu, you can run this:



# add a repository for stable releases
sudo wget -q -O /usr/share/keyrings/grafana.key \
  https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" \
  | sudo tee -a /etc/apt/sources.list.d/grafana.list

# install promtail
sudo apt-get update && sudo apt-get install -y promtail

By default, the configuration file /etc/promtail/config.yml will set up Promtail to ship logs to the URL: http://localhost:3100/loki/api/v1/push. This configuration will function correctly if you are using Vagrant with Virtualbox to run Loki. However, if you are not running the Loki server locally, you will need to modify the address (currently set to localhost) in the configuration accordingly.

As an option you can modify the configuration with the following:



server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      host: my_workstation
      agent: promtail
      __path__: /var/log/*log

And later restart the service:



sudo systemctl restart promtail

Ultimate EKS Baseline Cluster: Part 2 - Storage

Joaquin Menchaca — Mon, 24 Jul 2023 00:56:03 +0000

This may come as a surprise to some, but the AWS managed Kubernetes service, EKS (Elastic Kubernetes Service), no longer comes with support for storage through Kubernetes persistent volumes, which is required for many applications, such as databases.

You will need to add a (CSI) driver to enable persistent volumes support.

This guide, Part 2 of Ultimate EKS Baseline Cluster series, covers doing just that by installing CSI) called AWS EBS CSI Driver.

0. Prerequisites

These are are some prerequisites and initial steps needed to get started before provisioning a Kubernetes cluster and installing add-ons.

0.1 Knowledge: Systems

Basic concepts of systems, such as Linux and the shell (redirection, pipes, process substitution, command substitution, environment variables), as well as virtualization and containers are useful. The concept of a service (daemon) is important.

0.2 Knowledge: Kubernetes

In Kubernetes, familiarity with service types: ClusterIP, NodePort, LoadBalancer, ExternalName and the ingress resource are important.

Exposure to other types of Kubernetes resource objects used in this guide are helpful: persistentvolumeclaims (pvc), storageclass (sc), pods, deployments, statefulsets (sts), configmaps, serviceaccount (sa) and networkpolicies.

0.3 Tools

These are the tools used in this article series:

AWS CLI [aws] is a tool that interacts with AWS.
kubectl client [kubectl] a the tool that can interact with the Kubernetes cluster. This can be installed using adsf tool.
eksctl [eksctl] is the tool that can provision EKS cluster as well as supporting VPC network infrastructure.
POSIX Shell [sh] such as bash[bash] or zsh [zsh] are used to run the commands.

These tools are highly recommended:

adsf [adsf] is a tool that installs versions of popular tools like kubectl.
jq [jq] is a tool to query and print JSON data
GNU Grep [grep] supports extracting string patterns using extended Regex and PCRE.

0.3 Setup Environment Variables

These environment variables will be used throughout this guide. If opening up a new browser tab, make sure to set the environment variables accordingly.

# variables used to create EKS
export AWS_PROFILE="my-aws-profile" # CHANGEME
export EKS_CLUSTER_NAME="my-unique-cluster-name" # CHANGEME
export EKS_REGION="us-west-2"
export EKS_VERSION="1.26"

# KUBECONFIG variable
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# account id
export ACCOUNT_ID=$(aws sts get-caller-identity \
  --query "Account" \
  --output text
)

# ebs-csi-driver
export ROLE_NAME_ECSI="${EKS_CLUSTER_NAME}_EBS_CSI_DriverRole"
export ACCOUNT_ROLE_ARN_ECSI="arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME_ECSI"
POLICY_NAME_ECSI="AmazonEBSCSIDriverPolicy" # preinstalled by AWS
export POLICY_ARN_ECSI="arn:aws:iam::aws:policy/service-role/$POLICY_NAME_ECSI"

0.4 AWS Setup

There's an assumption that AWS CLI have been setup and configured with a profile. This is required before creating or interacting with an EKS cluster.

You can test access to a configured profile with the following command:

export AWS_PROFILE="<your-profile-goes-here>"
aws sts get-caller-identity

This should show something like the following with values appropriate to your environment, e.g. example output to IAM user named kwisatzhaderach:

{
    "UserId": "AIDAXXXXXXXXXXXXXXXXX",
    "Account": "XXXXXXXXXXXX",
    "Arn": "arn:aws:iam::XXXXXXXXXXXX:user/kwisatzhaderach"
}

0.5 EKS Cluster

This article requires that an EKS cluster has been previously provisioned using eksctl tool.

I wrote a previous article that covered this how to provision EKS with two commands:

Ultimate EKS Baseline Cluster: Part 1 - Provision EKS

0.5.1 Existing EKS Cluster

If you have an existing EKS cluster, but need to configure KUBECONFIG for access, you can run this:

mkdir -p $HOME/.kube # conditionally create ~/.kube
# use consistent $KUBECONFIG
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# update config pointed to by $KUBECONFIG 
aws eks update-kubeconfig \
  --name $CLUSTER \
  --region $REGION \
  --profile $PROFILE

0.5.2 Create a new EKS Cluster

If you have not setup an EKS cluster, you can set it up with the following commands (~20 minutes process):

mkdir -p $HOME/.kube # conditionally create ~/.kube
# use consistent $KUBECONFIG
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# provision EKS + add config to $KUBECONFIG 
eksctl create cluster \
  --version $EKS_VERSION \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME \
  --nodes 3

# setup OIDC provider for least privilege
eksctl utils associate-iam-oidc-provider \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --approve

Details of this were covered in the previous article

0.6 Kubernetes Client Setup

If you use asdf to install kubectl, you can get the latest version with the following:

# install kubectl plugin for asdf
asdf plugin-add kubectl \
  https://github.com/asdf-community/asdf-kubectl.git
asdf install kubectl latest

# fetch latest kubectl 
asdf install kubectl latest
asdf global kubectl latest

# test results of latest kubectl 
kubectl version --short --client 2> /dev/null

This should show something like:

Client Version: v1.27.4
Kustomize Version: v5.0.1

1. AWS EBS CSI driver

The current versions of EKS starting with 1.23 no longer come with a persistent volume support, so you have to install it on your own. The best method or at least the easiest way to install this, is using EKS add-ons facility. This will install the EBS CSI driver.

Installation of this component will require the following steps:

Create IAM Role (e.g. EBS_CSI_DriverRole) and associate it to Kubernetes service account (i.e. ebs-csi-controller-sa).
Deploy AWS EBS CSI driver using EKS add-ons facility, which also sets up the Kubernetes service account (i.e. ebs-csi-controller-sa) with an association back to the above IAM Role (e.g. EBS_CSI_DriverRole).
Create storage class that uses new the EBS CSI driver

1.1 Setup IAM Role and K8S SA association

The following process will create an IAM Role with permissions to access AWS EBS API. The service account ebs-csi-controller-sa will be created later when installing the driver.

# AWS IAM role bound to a Kubernetes service account
eksctl create iamserviceaccount \
  --name "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --attach-policy-arn $POLICY_ARN_ECSI \
  --role-only \
  --role-name $ROLE_NAME_ECSI \
  --approve

This will create a IAM Role, which you can verify with:

aws iam get-role --role-name $ROLE_NAME_ECSI

This should show something like this:

{
    "Role": {
        "Path": "/",
        "RoleName": "mycluster_EBS_CSI_DriverRole",
        "RoleId": "AROAZYKZFDW7YZGW3Q5S7",
        "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/mycluster_EBS_CSI_DriverRole",
        "CreateDate": "2023-07-07T21:19:32+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D"
                    },
                    "Action": "sts:AssumeRoleWithWebIdentity",
                    "Condition": {
                        "StringEquals": {
                            "oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D:aud": "sts.amazonaws.com",
                            "oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
                        }
                    }
                }
            ]
        },
        "Description": "",
        "MaxSessionDuration": 3600,
        "Tags": [
            {
                "Key": "alpha.eksctl.io/cluster-name",
                "Value": "mycluster"
            },
            {
                "Key": "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
                "Value": "mycluster"
            },
            {
                "Key": "alpha.eksctl.io/iamserviceaccount-name",
                "Value": "kube-system/ebs-csi-controller-sa"
            },
            {
                "Key": "alpha.eksctl.io/eksctl-version",
                "Value": "0.141.0-dev+5c8318ed5.2023-05-12T11:33:48Z"
            }
        ],
        "RoleLastUsed": {}
    }
}

1.2 Install AWS EBS CSI Drvier

This installation uses the EKS Addons feature to install the component.

# Install Addon
eksctl create addon \
  --name "aws-ebs-csi-driver" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --service-account-role-arn $ACCOUNT_ROLE_ARN_ECSI \
  --force

# Pause here until driver is active
ACTIVE=""; while [[ -z "$ACTIVE" ]]; do
  if eksctl get addon \
       --name "aws-ebs-csi-driver" \
       --region $EKS_REGION \
       --cluster $EKS_CLUSTER_NAME \
    | tail -1 \
    | awk '{print $3}' \
    | grep -q "ACTIVE"
  then
    ACTIVE="1"
  fi
done

It is important to wait until status changes to ACTIVE before proceeding.

You can inspect the pods created by running the following command:

kubectl get pods \
  --namespace "kube-system" \
  --selector "app.kubernetes.io/name=aws-ebs-csi-driver"

This should show something like:

NAME                                  READY   STATUS    RESTARTS   AGE
ebs-csi-controller-6d5b7bfd56-bwr5x   6/6     Running   0          2m1s
ebs-csi-controller-6d5b7bfd56-wtxf6   6/6     Running   0          2m2s
ebs-csi-node-hjmf5                    3/3     Running   0          2m2s
ebs-csi-node-tzpgs                    3/3     Running   0          2m2s

You can verify the service account annotations references the IAM Role for the EBS CSI driver.

kubectl get serviceaccount "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --output yaml

This should show something like the following

apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXX:role/my-unique-cluster-name_EBS_CSI_DriverRole
  creationTimestamp: "2023-07-24T21:12:02Z"
  labels:
    app.kubernetes.io/component: csi-driver
    app.kubernetes.io/managed-by: EKS
    app.kubernetes.io/name: aws-ebs-csi-driver
    app.kubernetes.io/version: 1.21.0
  name: ebs-csi-controller-sa
  namespace: kube-system
  resourceVersion: "1922"
  uid: 1b8ecfdb-ce64-491c-8ced-e08b5519755c

1.2.1 Sidebar: eks-addons vs helm chart?

The CSI driver can be installed using either EKS Addons facility from the previous step, or using the aws-ebs-csi-driver Helm chart. I prefer the EKS addons because of simplicity, but also because it installs an extra snapshot container that doesn't come by default with the helm chart.

I documented the full process using the helm chart in this article:

EKS + EBS Storage with eksctl

1.3 Create storage class that uses the EBS CSI driver

In order to use the driver, we will need to create a storage class. You can do so by running the following command:

# create ebs-sc storage class
cat <<EOF | kubectl apply --filename -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF

When completed you can verify the storage class was created with:

kubectl get storageclass

This should show something like this:

NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-sc          ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   17s
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  30m

1.4 Set new storage class to the default (optional)

This is an optional step. As there’s no functional default storage class, we can set the newly created storage class to be the default with the following commands:

kubectl patch storageclass gp2 --patch \
 '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass ebs-sc --patch \
 '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

After this, you can verify the change with:

kubectl get storageclass

This should show something like:

NAME               PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-sc (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   31s
gp2                kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  30m

1.5 Testing Persistent Volume

In this small test, we deploy a pod that continually writes to the external volume, and a volume claim to allocate storage using the storage class we created earlier.

If this works, the storage will be provisioned in the cloud to create the volume, and then it will be attached to the node and mounted in the pod. If this fails, you will see that the pod will be stuck in pending mode.

# create pod with persistent volume
kubectl create namespace "ebs-test"

# deploy application with mounted volume
cat <<-'EOF' | kubectl apply --namespace "ebs-test" --filename -
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: ubuntu
      command: ["/bin/sh"]
      args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
      volumeMounts:
      - name: persistent-storage
        mountPath: /data
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: ebs-claim
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 4Gi
EOF

You can test the results of the volume creation with the following command:

kubectl get all,pvc --namespace "ebs-test"

We can also look at the events that took place in this namespace with:

kubectl events --namespace "ebs-test"

This will show something like this:

LAST SEEN           TYPE      REASON                   OBJECT                            MESSAGE
56s                 Warning   FailedScheduling         Pod/app                           0/3 nodes are available: persistentvolumeclaim "ebs-claim" not found. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..
55s                 Normal    WaitForPodScheduled      PersistentVolumeClaim/ebs-claim   waiting for pod app to be scheduled
54s                 Normal    Provisioning             PersistentVolumeClaim/ebs-claim   External provisioner is provisioning volume for claim "ebs-test/ebs-claim"
54s (x2 over 54s)   Normal    ExternalProvisioning     PersistentVolumeClaim/ebs-claim   waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
50s                 Normal    Scheduled                Pod/app                           Successfully assigned ebs-test/app to ip-192-168-22-207.us-west-2.compute.internal
50s                 Normal    ProvisioningSucceeded    PersistentVolumeClaim/ebs-claim   Successfully provisioned volume pvc-39b9cf94-8b35-436c-b56d-e2fa587245ee
48s                 Normal    SuccessfulAttachVolume   Pod/app                           AttachVolume.Attach succeeded for volume "pvc-39b9cf94-8b35-436c-b56d-e2fa587245ee"
44s                 Normal    Pulling                  Pod/app                           Pulling image "ubuntu"
41s                 Normal    Pulled                   Pod/app                           Successfully pulled image "ubuntu" in 2.952872938s (2.952907339s including waiting)
41s                 Normal    Created                  Pod/app                           Created container app
41s                 Normal    Started                  Pod/app                           Started container app

1.6 Delete test application

kubectl delete pod app --namespace "ebs-test"
kubectl delete pvc ebs-claim --namespace "ebs-test"
kubectl delete ns "ebs-test"

2.0 Deleting EKS

When deleting up the EKS cluster, you may want to run through these steps.

2.1 Deleting persistent volume claims

When deleting EKS cluster, if you did not delete persistent volumes, you will have left over unused EBS volumes eating costs.

You should delete all the persistent volume claims, which will delete associate persistent volumes. You can list all of the persistentvolumeclaim resources with the following command:

kubectl get persistentvolumeclaim \
  --all-namespaces | grep -v none

Not that some of these will not be deleted if there is an application running that has a lock to the storage. So you will need to delete the associated application as well.

2.2 Reset Default to original Storage Class

As a precaution, we don’t want to have any resources locked that may prevent deletion of the Kubernetes cluster. Run this command if we changed the defaults earlier.

kubectl patch storageclass ebs-sc --patch \
  '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass gp2 --patch \
  '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

2.3 IAM Roles

These should be removed when removing Kubernetes with eksctl, but it is a good practice to remove them just in case.

eksctl delete iamserviceaccount \
  --name "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION

2.4 Kubernetes cluster

Finally, the Kubernetes cluster itself can be deleted.

eksctl delete cluster \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME

Conclusion

This second article shows add support for storage, called persistent volumes on Kubernetes.

The automation using the eksctl will do the following additional things besides provisioning EKS:

setup restricted access to AWS cloud resources using IRSA, which associates KSA with IAM Role using an OIDC provider.
install applications using EKS addons facility, specifically the EBS CSI driver.

In future articles, I will cover how to add support for load balancers and network policies, as well as a demo application that demonstrates all of these features.

Ultimate EKS Baseline Cluster: Part 1 - Provision EKS

Joaquin Menchaca — Fri, 21 Jul 2023 19:14:52 +0000

The AWS managed Kubernetes service, EKS (Elastic Kubernetes Service), has the highest level of complexity amongst cloud offerings.

Besides having to build out the networking, routing, security, and worker nodes separately from the managed master nodes, there’s no longer any bundled support for storage starting from 1.23, and the existing support for external load balancers is limited.

This tutorial, presented in separate parts, will demonstrate how to build out a robust Kubernetes cluster with just a few commands.

EKS Cluster with support for OIDC
Add Storage support: persistent volumes using EBS
Add load balancer support: layer 4 (NLB) and layer 7 (ALB)
Add traffic security support: network policies using Calico
Demo application with Dgraph putting it all together

In order to secure services that need access to cloud resources, an OIDC provider through IAM Roles for Service Account facility will be used.

0. Prerequisites

These are are some prerequisites and initial steps needed to get started before provisioning a Kubernetes cluster and installing add-ons.

0.1 Knowledge: Systems

0.2 Knowledge: Networking

This article requires some basic understanding of networking with TCP/IP and the OSI model, specifically the Transport Layer 4 and Application Layer 7 for HTTP. This article covers using load balancing and reverse proxy.

In Kubernetes, familiarity with service types: ClusterIP, NodePort, LoadBalancer, ExternalName and the ingress resource are important.

0.4 Tools

These are the tools used in this article series:

AWS CLI [aws] is a tool that interacts with AWS.
kubectl client [kubectl] a the tool that can interact with the Kubernetes cluster. This can be installed using adsf tool.
helm [helm] is a tool that can install Kubernetes applications that are packaged as helm charts.
eksctl [eksctl] is the tool that can provision EKS cluster as well as supporting VPC network infrastructure.
POSIX Shell [sh] such as bash[bash] or zsh [zsh] are used to run the commands.

These tools are highly recommended:

adsf [adsf] is a tool that installs versions of popular tools like kubectl.
jq [jq] is a tool to query and print JSON data
GNU Grep [grep] supports extracting string patterns using extended Regex and PCRE.

0.5 AWS Setup

Before getting started on EKS, you will need to set up billing to an AWS account (there’s a free tier), and then configure a profile that has provides access to an IAM User identity. See Setting up the AWS CLI for more information on configuring a profile.

After setup, you can test access with the following below:

export AWS_PROFILE="<your-profile-goes-here>"
aws sts get-caller-identity

This should show something like the following with values appropriate to your environment, e.g. example output to IAM user named kwisatzhaderach:

0.6 Kubernetes Client Setup

If you use asdf to install kubectl, you can get the latest version with the following:

# install kubectl plugin for asdf
asdf plugin-add kubectl \
  https://github.com/asdf-community/asdf-kubectl.git
asdf install kubectl latest

# fetch latest kubectl 
asdf install kubectl latest
asdf global kubectl latest

# test results of latest kubectl 
kubectl version --short --client 2> /dev/null

This should show something like:

Client Version: v1.27.1
Kustomize Version: v5.0.1

Also, create directory to store Kubernetes configurations that will be used by the KUBECONFIG env variable:

mkdir -p $HOME/.kube

0.7 Setup Environment Variables

These environment variables will be used throughout this guide. If opening up a new browser tab, make sure to set the environment variables accordingly.

# variables used to create EKS
export AWS_PROFILE="my-aws-profile" # CHANGEME
export EKS_CLUSTER_NAME="my-unique-cluster-name" # CHANGEME
export EKS_REGION="us-west-2"
export EKS_VERSION="1.26"
# KUBECONFIG variable
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# account id
export ACCOUNT_ID=$(aws sts get-caller-identity \
  --query "Account" \
  --output text
)

1. Provision an EKS cluster

After prerequisite tools are installed and setup, we can start provisioning cloud resources and deploy components to Kubernetes.

1.1. Provision the EKS Cluster

The cluster can be brought up with the following command:

eksctl create cluster \
  --version $EKS_VERSION \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME \
  --nodes 3

Also, check the status of the worker nodes and applications running on Kubernetes.

kubectl get nodes
kubectl get all --all-namespaces

This should show something like the following.

1.2. Install the Kubernetes client

Once this finished in about 20 minutes to provision both VPC and EKS, install a kubectl version that matches the Kubernetes server version:

# fetch exact version of Kubernetes server (Requires GNU Grep)
VER=$(kubectl version --short 2> /dev/null \
  | grep Server \
  | grep -oP '(\d{1,2}\.){2}\d{1,2}'
)

# setup kubectl tool
asdf install kubectl $VER
asdf global kubectl $VER

⚠️ NOTE: The above command requires GNU grep. If you have Homebrew, you can run brew install grep. Windows can get this with MSYS2 or git-bash.

1.3 Add OIDC Provider Support

The EKS cluster has an OpenID Connect (OIDC) issuer URL associated with it. To use IRSA (IAM roles for service accounts), an IAM OIDC provider must exist for the cluster’s OIDC issuer URL.

You can set this up with the following command:

eksctl utils associate-iam-oidc-provider \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --approve

You can verify the OIDC provider is added with the following:

OIDC_ID=$(aws eks describe-cluster \
  --name $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --query "cluster.identity.oidc.issuer" \
  --output text \
  | cut -d '/' -f 5
)

aws iam list-open-id-connect-providers \
  | grep $OIDC_ID \
  | cut -d '"' -f4 \
  | cut -d '/' -f4

Resources

AWS Cloud Resources

These are some of the AWS resource objects used in this guide.

Conclusion

This first article shows have to get a modern Kubernetes cluster working with support for OIDC that will allow restricted access to cloud resources from Kubernetes applications using concept of least privilege

This article uses eksctl tool to provision the necessary cloud resources to bring up EKS. The eksctl tool will use Cloud Formation stacks to deploy the necessary cloud resources (VPC, EC2, SGs, etc) that make up the full solution.

I am sure there are many wondering, why not use something like Terraform for provisioning?

That is actually a good idea except that this will be more involved because we'll have to deploy all of the components required to make a functional EKS cluster: VPC, internet gateways, nat groups, subnet, routing tables, auto-scale groups, security groups, etc. in addition to the master control plane (EKS).

In the future, I would like to make a similar series with Terraform to setup the base EKS cluster, and then use Terraform with templatefile or Helmfile to manage installing helm charts and kubernetes manifests.

The ultimate goal of this series is to quickly get a robust and secure Kubernetes cluster with necessary features like storage and networking with preferably low complexity and friction.

From here, we can explore other developments and tutorials on Kubernetes, such as o11y or observability (PLG, ELK, ELF, TICK, Jaeger, Pyroscope), service mesh (Linkerd, Istio, NSM, Consul Connect, Cillium), and progressive delivery (ArgoCD, FluxCD, Spinnaker).

Install Terraform with tfenv

Joaquin Menchaca — Sun, 15 Jan 2023 18:36:35 +0000

There will come a time when you will need to change to different versions of Terraform. Similar to tools like rbenv for Ruby or pyenv for Python, there's a popular tool used to install different versions of Terraform called tfenv.

Most tooling for infrastructure operations runs on Intel processors, called the amd64 architecture with in Terraform, but given the recent addition of Macbook M1 laptops, the architecture is arm64. Older versions of Terraform that may be needed will require installing Terraform that is only available on Intel processors (amd64 binaries). This can be installed using tfenv as well.

This article will have cover the following sections:

Installing tfenv on Ubuntu (linux) and mac OS (darwin)
Installing a latest or recent version of Terraform
Installing legacy versions of Terraform on macOS with Arm processor (Macbook M1/M2)
Installing Legacy providers plugins (Ubuntu and Macbook M1/M2)

Installing tfenv

Here's how you can install tfenv on Ubuntu (linux) or mac OS (darwin).

Installing tfenv on Ubuntu

git clone https://github.com/tfutils/tfenv.git ${HOME}/.tfenv
export PATH="${HOME}/.tfenv/bin:${PATH}"
# install to appropriate shell startup file, e.g. $HOME/.bashrc
echo 'export PATH="$HOME/.tfenv/bin:$PATH"' >> ${HOME}/.profile

Installing tfenv on mac OS

brew install tfenv

Installing Terraform

Once tfenv is installed and available in the path, you can simply run something like this:

tfenv install 1.3.7

Using tfenv list-remote you can fetch different versions of Terraform that are available for installation. From there, you can use GNU grep to select the desired version. For example, here's how to install the latest version:

LATEST_VERSION=$(tfenv list-remote \
 | grep -vP '\d.\d{0,2}.\d{0,2}-.*' \
 | head -1
)

tfenv install ${LATEST_VERSION}

NOTE: On macOS, which has BSD flavor grep, it doesn't support PCRE. To get GNU Grep with support for PCRE, use brew install grep.

You can extract the last minor version by passing in an explicit major version to GNU Grep:

# find latest 0.12
LATEST_0_12=$(tfenv list-remote  \
 | grep -oP '^0.12.\d{1,2}' \
 | sort --version-sort \
 | tail -1
)

tfenv install ${LATEST_0_12}

For the latest version, whatever that is, you can also just run the following:

tfenv install latest

Using the desired Terraform version

When you want to use the a particular version of an installed version terraform, you can select it with the following:

tfenv use 1.3.7

Also, to have the Terraform switch automatically when navigating to the project directory, you can place a .terraform-version to auto-select the version:

mkdir -p  ${HOME}/my_project/.terraform-version
echo '1.3.7' > ${HOME}/my_project/.terraform-version

# change version of Terraform automatically
cd ${HOME}/my_project/.terraform-version

Installing Legacy x64 versions on Mac

If you Macbook sports a arm64 processor, you need to get Rosetta to run amd64 binaries on macOS. Afterward, you can follow these steps below to install legacy versions of terraform.

export TFENV_ARCH="amd64"

# install amd64 terrafrom version since there are no arm64 binaries
tfenv install 0.12.31
tfenv use 0.12.31

Installing Legacy providers

In recent versions of Terraform, this installation will happen automatically when you run terraform init

First, you need to find the appropriate plugin version that matches the supported version of Terraform. In the case of terraform-provider-kubectl, you need to install 1.11.2 of that version of that plugin.

Installing legacy provider on Ubuntu Linux

ARCH="linux_amd64"
VERS="1.11.2"
PKG="terraform-provider-kubectl_${VERS}_${ARCH}.zip"
REPO="gavinbunney/terraform-provider-kubectl"
URL="https://github.com/${REPO}/releases/download/v${VERS}/${PKG}"

# setup plugins directory if this does not exist already
mkdir -p ${HOME}/.terraform.d/plugins/${ARCH}/

# download archive and install plugin
pushd ${HOME}/Downloads
curl -sOL ${URL}
unzip ${PKG} && rm README.md LICENSE
mv terraform-provider-kubectl_v${VERS} \
  ${HOME}/.terraform.d/plugins/${ARCH}/
popd

Installing legacy provider on Macbook M1

For current Macbooks running the arm64 processor, you will need to install the legacy provider for amd64 (not arm64). This is how you can do that.

ARCH="darwin_amd64" # install amd64 plugin for legacy terraform 
VERS="1.11.2"
PKG="terraform-provider-kubectl_${VERS}_${ARCH}.zip"
REPO="gavinbunney/terraform-provider-kubectl"
URL="https://github.com/${REPO}/releases/download/v${VERS}/${PKG}"

# setup plugins directory if this does not exist already
mkdir -p $HOME/.terraform.d/plugins/${ARCH}/

# download archive
pushd ${HOME}/Downloads
curl -sOL ${URL}
unzip ${PKG} && rm README.md LICENSE
mv terraform-provider-kubectl_v${VERS} \
  ${HOME}/.terraform.d/plugins/${ARCH}/
popd

Conclusion

I hope this is useful to your Terraform adventures. The reason why locking down versions of Terraform is important, is that there are significant language changes that make the code incompatible. The installer dependency chain with Terraform modules and providers will need to match the target version of the terraform command.

In addition to this, the state that is generated from interacting with the provider will likely need to match, so it is important that the infrastructure that created with a particular version of Terraform, also be removed when ultimately terraform destroy is used.

As the infrastructure progresses forward, there are techniques you can use to smoothly transition. One method it to have modular modules that depend on looking up the state in the cloud through data sources, and important existing infrastructure into the state using terraform import. This is definitely material for a future article.

In the meantime, you can use the supported version with tools like tfenv. Another tool, as an alternative to tfenv is asdf. I have not explored supported legacy amd64 legacy binaries on Macbook M1 yet, but assume this can work. The asdf is nice in that it has a plugin architecture to support a variety of languages.

Lastly, about this article, which I am sure might unnerve shell purists, where I use $HOME instead of ~ and I use of curly braces for ${VARIABLES}, when they are not needed, when simply $VARIABLE is preferred: this was done intentionally to make the variables light up in color syntax highlighters on different platforms, and $HOME was used for novice uses that may not be familiar with ~ notation.

GKE with Consul Service Mesh

Joaquin Menchaca — Sun, 04 Dec 2022 01:05:02 +0000

This article shows how to set up and get started with CCSM (Consul Connect service mesh) or more recently called just Consul Service Mesh.

This article will cover how to install and configure services to use CCSM. An example application Dgraph, a distributed graph database, will be used as this demonstrates a real world application.

📔 NOTE: This was tested on following below 
and may not work if versions are significantly 
different.

* Kubernetes API v1.22
* gcloud 402.0.0
* gsutil 5.13
* kubectl v1.22
* kustomize v4.5.4
* helm v3.8.2
* helmfile v0.144.0
* Docker 20.10.17
* Dgraph v21.03.2
* Consul 1.13.2

About Consul

Consul is a popular tool for service discovery and a key-value store that was released in April-2014.

Service discovery is important for clusters members or microservices as it provides “automatic detection of devices and services offered by these devices on a computer network (ref)”. This allows “applications and microservices locate different components on a network (ref)”.

The key-value store is a network database to store hash maps (also called associative arrays or dictionaries). This allows services create and retrieve configuration.

Building upon this, Hashicorp developed consul-template, which is essentially cloud native change configuration, and Consul Connect, now called Consul Service Mesh, which can automatically inject side-car proxy containers into your network, so that services can communicate securely.

About Service Mesh

A service mesh uses automation to secure internal network traffic between member nodes. It does this by inserting reverse-proxy sidecar containers to every pod that is apart of the service mesh.

With a network of side-car proxies installed, the traffic can be further secured using strict mTLS, where not only the client must authenticate the validity of the server, but the server must authenticate the validity of the client. This is on top of the encryption of traffic between all the members of the mesh.

A service mesh is divided into three planes (illustration below): control plane to manage the overall service mesh, a data plane that consists of the members within the mesh that are secured with a proxy, and observability plane to monitor traffic from within the mesh.

📔 NOTE: Observability is not supported for Consul Service 
Mesh with services that use multiple-ports.

Consul Connect leverages off of Consul to manage the connectivity through use of service discovery, health checks, and a service catalog. Envoy is the default proxy that is injected into each of the pods to create the service mesh. This proxy can be swapped for another proxy component, such as HAProxy or NGINX.

Requirements

These are the requirements to use this solution.

Accounts

No commercial licenses are needed for either Consul and Dgraph. All of the tools are accessible from the public Internet. For creating resources on Google Cloud, you will need to create an account.

Google Cloud account with ownership of a project where you can deploy resources (where billing account was linked to the project)

Knowledge

You should be familiar or have exposure to the following concepts to get more thorough understanding of this tutorial:

virtual hosting, reverse-proxy, and load balancer
Layer 4 vs Layer 7 (OSI), TCP/IP, routing, gateway
HTTP/1.1 vs HTTP/2 and gRPC
images vs containers
virtual machines instances (nodes) vs containers vs pods

For Kubernetes, experience with deploying applications with service resources is useful, but even if you don’t have this, this guide will walk you through it. Configuring KUBECONFIG to access the Kubernetes cluster with Kubernetes client (kubectl) and using Helm (helm), so familiarity to this is useful.

For Google Cloud, you should be familiar Google Cloud SDK (gcloud tool) with setting up an account, project, and provisioning resources. This is important as there are cost factors involved in setting these things up.

Tools (Required)

Google Cloud SDK (gcloud command) to interact with Google Cloud
Kubernetes client (kubectl command) to interact with Kubernetes
Helm (helm command) to install Kubernetes packages
helm-diff plugin to see differences about what will be deployed.
helmfile (helmfile command) to automate installing many helm charts
Kustomize (kustomize command) to apply patches to existing Helm charts

Tools (Recommended)

These tools are useful in using the automation used form within this article.

POSIX shell (sh) such as GNU Bash (bash) or Zsh (zsh): these scripts in this guide were tested using either of these shells on macOS and Ubuntu Linux.
GNU stream-editor (sed) and GNU grep (grep): scripts were tested with these tools. Note that BSD versions of these tools may NOT WORK, such as tools bundled with macOS or BSD.
Docker Engine (docker command) to automate building and pushing running pydgraph client to Google container registry.
git (git command) to download source code from git code repositories.

Project Setup

This will setup all the content for this tutorial.

Directory Structure

The directory structure should look like this:

~/projects/consul_connect
├── consul
│   └── helmfile.yaml
└── examples
    └── dgraph
        ├── helmfile.yaml
        └── pydgraph_client.yaml

In GNU Bash, you can create the above structure like this:

export PROJECT_DIR=~/projects/consul_connect
mkdir -p $PROJECT_DIR/{examples/dgraph,consul}
cd $PROJECT_DIR
touch {consul,examples/dgraph}/helmfile.yaml \
 examples/dgraph/pydgraph_client.yaml

Environment Variables

These environment variables will be used in this project. Create a file called env.sh with the contents below, changing values as appropriate, and then run source env.sh.

# gke
export GKE_PROJECT_ID="my-gke-project" # CHANGE ME
export GKE_CLUSTER_NAME="csm-demo"
export GKE_REGION="us-central1"
export GKE_SA_NAME="gke-worker-nodes-sa"
export GKE_SA_EMAIL="$GKE_SA_NAME@${GKE_PROJECT_ID}.iam.gserviceaccount.com"
export KUBECONFIG=~/.kube/$GKE_REGION-$GKE_CLUSTER_NAME.yaml

# other
export USE_GKE_GCLOUD_AUTH_PLUGIN=True
export ClOUD_BILLING_ACCOUNT="<my-billing-account>" # CHANGEME

Google Project Setup

For this tutorial, we’ll need to setup a Google cloud project and provide access to allow use to create the necessary cloud resources. Here is an example of how you can set this up with gcloud:

# enable billing and APIs for GKE project if not done already
gcloud projects create $GKE_PROJECT_ID
gcloud config set project $GKE_PROJECT_ID
gcloud beta billing projects link $GKE_PROJECT_ID \
  --billing-account $ClOUD_BILLING_ACCOUNT
gcloud services enable "container.googleapis.com"

Provision Cloud Resources

These instructions will create the necessary cloud resources for this project.

Provision Google Kubernetes Engine cluster

The steps below will allow you to bring up a Kubernetes cluster with 3 worker nodes.

📔 NOTE: This will deploy a robust 3 worker node Kubernetes 
cloud that is suitable for Consul.  This will create a 
principal identity (Google Service Account) with the minimal 
necessary privileges required to manage the Kubernetes nodes 
(GCE).

📔 NOTE: For production environments, you will want to 
explore further security measures, such as private cluster, 
to block access from the public Internet.

source env.sh

#######################
# GSA with least priv for GKE
##########################################
ROLES=(
  roles/logging.logWriter
  roles/monitoring.metricWriter
  roles/monitoring.viewer
  roles/stackdriver.resourceMetadata.writer
)

gcloud iam service-accounts create $GKE_SA_NAME \
 --display-name $GKE_SA_NAME --project $GKE_PROJECT_ID

# assign google service account to roles in GKE project
for ROLE in ${ROLES[*]}; do
 gcloud projects add-iam-policy-binding $GKE_PROJECT_ID \
  --member "serviceAccount:$GKE_SA_EMAIL" \
  --role $ROLE
done

#######################
# GKE with least priv. GSA + Workload Identity
##########################################
gcloud container clusters create $GKE_CLUSTER_NAME \
  --project $GKE_PROJECT_ID \
  --region $GKE_REGION \
  --num-nodes 1 \
  --service-account "$GKE_SA_EMAIL" \
  --machine-type "e2-standard-2" \
  --enable-ip-alias \
  --workload-pool "$GKE_PROJECT_ID.svc.id.goog"

#######################
# KUBECONFIG
##########################################
gcloud container clusters get-credentials $GKE_CLUSTER_NAME \
  --project $GKE_PROJECT_ID \
  --region $GKE_REGION

You can test access to the cluster as well as the components installed with the following commands:

kubectl get nodes
kubectl get all --all-namespaces

Another useful command to test a new cluster is to see how many resources are available and what is consumed in the new cluster:

kubectl top nodes
kubectl top pods --all-namespaces

Deploy Kubernetes Resources

This section covers deploying Kubernetes resources such as Deployment, StatefulSet, ServiceAccount, Service, and so on. This will cover installing the Consul Connect service mesh, Dgraph, and pydgraph-client to access Dgraph through the service mesh.

Deploy Consul Connect service mesh

This will deploy the Consul Connect Service mesh. Save the following code below as consul/helmfile.yaml:

repositories:
  # https://artifacthub.io/packages/helm/hashicorp/consul
  - name: hashicorp
    url: https://helm.releases.hashicorp.com

releases:
  - name: consul
    namespace: consul
    chart: hashicorp/consul
    version: 0.49.0
    values:
      - global:
          name: consul
          datacenter: dc1
          {{- if eq (env "CCSM_SECURITY_ENABLED") "true" }}
          tls:
            enabled: true
            enableAutoEncrypt: true
            verify: true
          gossipEncryption:
            autoGenerate: true
          acls:
            manageSystemACLs: true
          {{- end }}
        server:
          securityContext:
            runAsNonRoot: false
            runAsUser: 0
        connectInject:
          enabled: true
        controller:
          enabled: true
        ui:
          enabled: true

This Helm chart configuration values will install Consul Connect service mesh with automatic injection enabled. When you deploy a pod with annotation of consul.hashicorp.com/connect-inject: "true", side-car containers will be installed to copy the consul binary into the container and setup and configure Envoy proxy. The service proxy resources will be used as a blueprint to register the service with Consul's service catalog and configure the Envoy proxy.

Run the following to deploy the service mesh:

source env.sh
helmfile --file ./consul/helmfile.yaml apply

You can check that everything is deployed with:

kubectl get all --namespace consul

This should show something like this:

Deploy Observability

Currently observability is not supported with multi-port services like Dgraph. Hopefully this will get fixed in the future.

For further information, see:

Deploy Dgraph

Dgraph is a distributed graph database communicates through both HTTP on port 8080 and gRPC on port 9080. Dgraph uses the DQL (Dgraph Query Language) through either gRPC or HTTP, and can also use GraphQL with HTTP. Dgraph supports administrative operations using GraphQL or REST.

For this reason, to fully use Dgraph on a service mesh, you have to use the recently added multi-port configuration with Consul Connect. This requires separating the single multi-port service proxy into two separate service proxies: one for gRPC (9080) and one for HTTP (8080).

Save the following helmfile config below as examples/dgraph/helmfile.yaml:

repositories:
  # https://artifacthub.io/packages/helm/dgraph/dgraph/0.0.19
  - name: dgraph
    url: https://charts.dgraph.io
  # https://artifacthub.io/packages/helm/main/raw
  - name: bedag
    url: https://bedag.github.io/helm-charts/

releases:
  # Dgraph additional resources required to support Consul
  - name: dgraph-extra
    chart: bedag/raw
    namespace: dgraph
    version:  1.1.0
    values:
      - resources:
          - apiVersion: v1
            kind: ServiceAccount
            metadata:
              name: dgraph-dgraph-zero

          - apiVersion: v1
            kind: ServiceAccount
            metadata:
              name: dgraph-dgraph-alpha

          - apiVersion: v1
            kind: ServiceAccount
            metadata:
              name: dgraph-dgraph-alpha-grpc

          - apiVersion: v1
            kind: Service
            metadata:
              name: dgraph-dgraph-alpha-grpc
            spec:
              ports:
              - name: grpc-alpha
                port: 9080
              publishNotReadyAddresses: true
              selector:
                app: dgraph
                chart: dgraph-0.0.19
                component: alpha
                release: dgraph
              type: ClusterIP

  # Dgraph cluster with 2 x StatefulSet (3 Zero pods, 3 Alpha pods)
  - name: dgraph
    namespace: dgraph
    chart: dgraph/dgraph
    version: 0.0.19
    needs:
      - dgraph/dgraph-extra
    values:
      - image:
          tag: v21.03.2
        zero:
          extraAnnotations:
            consul.hashicorp.com/connect-inject: 'true'
            # disable transparent-proxy for multi-port services
            consul.hashicorp.com/transparent-proxy: 'false'
            consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: "5080,7080"
            consul.hashicorp.com/transparent-proxy-exclude-outbound-ports: "5080,7080"
        alpha:
          extraAnnotations:
            consul.hashicorp.com/connect-inject: 'true'
            # disable transparent-proxy for multi-port services
            consul.hashicorp.com/transparent-proxy: 'false'
            # use these registered consul services for different ports
            consul.hashicorp.com/connect-service: 'dgraph-dgraph-alpha,dgraph-dgraph-alpha-grpc'
            consul.hashicorp.com/connect-service-port: '8080,9080'
            consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: "5080,7080"
            consul.hashicorp.com/transparent-proxy-exclude-outbound-ports: "5080,7080"
          configFile:
            config.yaml: |
              security:
                whitelist: {{ env "DG_ACCEPT_LIST" | default "0.0.0.0/0" | quote }}
    # patch existing resources using merge patches
    strategicMergePatches:
      # add serviceAccountName to Alpha StatefulSet
      - apiVersion: apps/v1
        kind: StatefulSet
        metadata:
          name: dgraph-dgraph-alpha
        spec:
          template:
            spec:
              serviceAccountName: dgraph-dgraph-alpha

      # add serviceAccountName to Zero StatefulSet
      - apiVersion: apps/v1
        kind: StatefulSet
        metadata:
          name: dgraph-dgraph-zero
        spec:
          template:
            spec:
              serviceAccountName: dgraph-dgraph-zero

      # add label to Alpha headless service
      - apiVersion: v1
        kind: Service
        metadata:
          name: dgraph-dgraph-alpha-headless
          labels:
            consul.hashicorp.com/service-ignore: 'true'

      # add label to Zero headless service
      - apiVersion: v1
        kind: Service
        metadata:
          name: dgraph-dgraph-zero-headless
          labels:
            consul.hashicorp.com/service-ignore: 'true'

    # patch existing resource using jsonPatches
    jsonPatches:
      # remove existing grpc port from serivce
      - target:
          version: v1
          kind: Service
          name: dgraph-dgraph-alpha
        patch:
          - op: remove
            path: /spec/ports/1

This helmfile config uses some advance features to make some necessary changes required by Consul Connect:

pre-install service accounts and new gRPC service all packaged up as dgraph-extras chart
render Dgraph resources with required annotations for consul
apply patches to add Dgraph headless service labels that instructs Consul to ignore these services when is configures the proxies.
remove gRPC port (9080) from the Dgraph Alpha service, as this was defined earlier as a separate gRPC service with the dgraph-extras chart.

Consul Connect will inject Envoy sidecar proxy containers. Dgraph Zero will get a sidecar for port 6080, while Dgraph Alpha will have two sidecar proxy containers per pod: one for gRPC at port 9080 and another one for HTTP at port 8080.

When ready to deploy all of this, run the following command:

source env.sh
helmfile --file ./examples/dgraph/helmfile.yaml apply

You can check on the status using:

kubectl get all --namespace dgraph

This should show something like:

You notice the extra containers per pod in the ready state, which are the Envoy proxy sidecar containers.

Deploy Pydgraph client

The client is a small python script that can load data into Dgraph using gRPC, and the container also has some useful tools like curl, grpcurl, and jq.

Save the following below as examples/dgraph/pydgraph_client.yaml:

repositories:
  # https://artifacthub.io/packages/helm/main/raw
  - name: bedag
    url: https://bedag.github.io/helm-charts/

releases:
  - name: pydgraph-client
    chart: bedag/raw
    namespace: pydgraph-client
    version:  1.1.0
    values:
      - resources:
          - apiVersion: v1
            kind: ServiceAccount
            metadata:
              name: pydgraph-client
          - apiVersion: apps/v1
            kind: Deployment
            metadata:
              name: pydgraph-client
            spec:
              replicas: 1
              selector:
                matchLabels:
                  app: pydgraph-client
              template:
                metadata:
                  labels:
                    app: pydgraph-client
                spec:
                  serviceAccountName: pydgraph-client
                  containers:
                  - name: pydgraph-client
                    image: {{ requiredEnv "DOCKER_REGISTRY" }}/pydgraph-client:{{ env "BUILD_VERSION" | default "latest" }}
                    ports:
                      - containerPort: 5000
                    env:
                      {{- if eq (env "CCSM_ENABLED") "true" }}
                      - name: DGRAPH_ALPHA_SERVER
                        value: localhost
                      - name: DGRAPH_GRPC_SERVER
                        value: localhost
                      {{- else }}
                      - name: DGRAPH_ALPHA_SERVER
                        value: {{ env "DGRAPH_RELEASE" | default "dgraph" }}-dgraph-alpha.{{ env "DGRAPH_NS" | default "dgraph" }}.svc.cluster.local
                      {{- end }}
                    resources:
                      requests:
                        memory: "64Mi"
                        cpu: "8m"
                      limits:
                        memory: "128Mi"
                        cpu: "25m"

          - apiVersion: v1
            kind: Service
            metadata:
              name: pydgraph-client
            spec:
              type: ClusterIP
              ports:
              - port: 80
                targetPort: 5000
              selector:
                app: pydgraph-client
    {{- if eq (env "CCSM_ENABLED") "true" }}
    strategicMergePatches:
      - apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: pydgraph-client
        spec:
          template:
            metadata:
              annotations:
                consul.hashicorp.com/connect-inject: "true"
                consul.hashicorp.com/transparent-proxy: "false"
                consul.hashicorp.com/connect-service-upstreams: >-
                  {{ env "DGRAPH_RELEASE" | default "dgraph" }}-dgraph-alpha:8080,{{ env "DGRAPH_RELEASE" | default "dgraph" }}-dgraph-alpha-grpc:9080
    {{- end }}

When ready to deploy this, you can run the following:

source env.sh
# https://hub.docker.com/r/darknerd/pydgraph-client
export DOCKER_REGISTRY=darknerd
export CCSM_ENABLED=true
helmfile --file ./examples/dgraph/pydgraph_client.yaml apply

You can check the deployment with the following:

kubectl get all --namespace pydgraph-client

This should result in something similar to the following:

Testing Upstream Traffic

Consul Connect will set up a tunnel between the upstream ports specified in the annotation to the ports that are serviced by Dgraph.

First remote into the client container:

CLIENT_NS="pydgraph-client"
PYDGRAPH_POD=$(kubectl get pods -n $CLIENT_NS --output name)
kubectl exec -ti -c "pydgraph-client" -n $CLIENT_NS \
  ${PYDGRAPH_POD} -- bash

One in the container, test that HTTP traffic is working:

curl --silent localhost:8080/health

For gRPC traffic, you can run the following:

grpcurl -plaintext -proto api.proto \
  localhost:9080 api.Dgraph/CheckVersion

Also, you can try loading data:

python3 load_data.py \
  --plaintext \
  --alpha localhost:9080 \
  --files ./sw.nquads.rdf \
  --schema ./sw.schema

These should work through the tunnel that is configured by Consul Connect using the Envoy proxy side-cars.

Dgraph Graphical Viewer: Ratel

Dgraph hosts an online graphical viewer at https://play.dgraph.io/. If you would like to access the data we deployed with load_data.py, you can run this in a new terminal tab:

kubectl port-forward svc/dgraph-dgraph-alpha -n dgraph 8080:8080

Now you can you can point the connection configuration in Ratel to http://localhost:8080:

Click on the Console and select Query and enter the following DQL:

{
 me(func:allofterms(name, "Star Wars")) @filter(ge(release_date, "1980")) {
   name
   release_date
   revenue
   running_time
   director {
    name
   }
   starring {
    name
   }
 }
}

Click Run to see the results of the query:

Consul User Interface

The Consul UI can be accessed by running this command in a new terminal tab:

source env.sh
kubectl port-forward service/consul-ui --namespace consul 8500:80

You can access the Consul UI through http://localhost:8500. The Consul UI should look like this below with other services appearing after Dgraph and pydgraph-client were deployed.

If you click on pydgraph-client, you can see the connections:

Cleanup

Kubernetes Resources

You can cleanup Kubernetes resources with the following:

source env.sh

# delete pydgraph-client
helmfile --file ./examples/dgraph/helmfile.yaml delete
kubectl delete namespace pydgraph-client

# delete dgraph
helmfile --file ./examples/dgraph/helmfile.yaml delete
kubectl delete pvc --selector app=dgraph --namespace "dgraph"
kubectl delete namespace dgraph

# delete consul
helmfile --file ./consul/helmfile.yaml delete
kubectl delete pvc --selector app=consul --namespace "consul"
kubectl delete namespace consul

It is important to delete the consul namespace if you intend to deploy new version of Consul Connect service mesh in the future. This is because there are secrets left behind that will break future installations, so deleting the namespace will avoid this scenario.

Cloud Resources

The Kubernetes cluster and the associated Google service account can be deleted with the following commands:

source env.sh

gcloud container clusters delete $GKE_CLUSTER_NAME \
  --project $GKE_PROJECT_ID --region $GKE_REGION

gcloud iam service-accounts delete $GKE_SA_EMAIL --project $GKE_PROJECT_ID

Addendum: Publishing Pygraph-Client Images

If you would like to publish the pydgraph-client images to an alternative registry, you can run the following steps below.
Download the source code

pushd examples
git clone \
  --depth 1 \
  --branch "consul" \
  git@github.com:darkn3rd/pydgraph-client.git
popd

Publishing to GCR

If you wish to use Google Container Registry, you can run the following.

pushd ./examples/pydgraph-client

###################
# STEP 1: Environment variables
#######################################
source env.sh
export GCR_PROJECT_ID="my-gcr-project"  # CHANGE ME
export DOCKER_REGISTRY="gcr.io/$GCR_PROJECT_ID"
export ClOUD_BILLING_ACCOUNT="<my-cloud-billing-account>" # CHANGEME

###################
# STEP 2: Create GCR project and enable GCR
# NOTE: Best practices is to use a single project for store/retreive images
#######################################
gcloud projects create $GCR_PROJECT_ID
gcloud config set project $GCR_PROJECT_ID
gcloud beta billing projects link $GCR_PROJECT_ID \
  --billing-account $ClOUD_BILLING_ACCOUNT
gcloud services enable "containerregistry.googleapis.com" # Enable GCR API
gcloud config set project $GKE_PROJECT_ID

###################
# STEP 3: Build local image
#######################################
docker build -t pydgraph-client:latest .

###################
# STEP 4: Publish Image to GCR
#######################################
gcloud auth configure-docker
docker tag pydgraph-client:latest $DOCKER_REGISTRY/pydgraph-client:latest
docker push $DOCKER_REGISTRY/pydgraph-client:latest

###################
# STEP 5: Grant GCR read access to GKE containers
#######################################
gsutil iam ch \
  serviceAccount:$GKE_SA_EMAIL:objectViewer \
  gs://artifacts.$GCR_PROJECT_ID.appspot.com

popd

Publising to DockerHub

If you have an account on DockerHub, you can publish it there with these steps:

pushd ./examples/pydgraph-client

###################
# STEP 1: Environment variables
#######################################
source env.sh
export DOCKER_REGISTRY="<your-docker-hub-account-goes-here>"

###################
# STEP 2: Build local image
#######################################
docker build -t pydgraph-client:latest .

###################
# STEP 3: Publish Image to DcokerHub
#######################################
docker login # IMPORTANT: use api-token not the actual password
docker tag pydgraph-client:latest $DOCKER_REGISTRY/pydgraph-client:latest
docker push $DOCKER_REGISTRY/pydgraph-client:latest

popd

Resources

These are some resources and references that may be useful in using this solution.

Consul Documentation

Gateways and Ingress

These are links for north-south traffic into mesh.
I have not tested these solutions yet

These are links that cover integration of either ingress controllers or API gateways with Consul. This may be using Consul as a backend database or the Consul Connect service mesh itself.

📔 NOTE: I have not tested the content of this material, just 
documenting any material I find on the topic for later 
exploration.  If you find any useful material out there, 
please send me a note.

Amabassador Edge Stack integration with Consul
Consul API Gateway
Using HashiCorp Consul with Kong Ingress Controller for Kubernetes
Getting Started With Traefik Proxy and HashiCorp Consul
Taefik Consul Provider configuration
traefik-consul walk-through
consul ingress controllers integration with Traefik or Kong using terrafom
Consul on EKS using nginx as ingress (transparent mode)

Tracing

Dgraph Documentation

DQL Fundamentals

Helmfile

https://github.com/helmfile/helmfile
What is Helmfile? by Paul Czarkowski (VMWare Tanzu Developer Center)

Blog Source Code

This is some code that I developed when testing Consul Connect service mesh solution.

Blog Source Code: https://github.com/darkn3rd/blog_tutorials/tree/master/kubernetes/gke/service-mesh/consul-connect
HTTP/gRPC Greeter Application: https://github.com/darkn3rd/greeter
pydgraph-client w consul support: https://github.com/darkn3rd/pydgraph-client/tree/consul

Conclusion

There you have it, a small (cough) overview how to get started with Consul Connect Service Mesh. In particular, here some of the takeaways:

Provisioning Kubenertes (GKE)
(addendum) Provisioning GCR and publishing images to GCR
Deploying Consul Connect Service Mesh on GKE
Deploying a server and a client with multiport support: HTTP and gRPC
Testing HTTP traffic with curl and gRPC traffic with gprcurl.
Limitations and Challenges with current multi-port scenarios

Additionally, here’s some extra takeaways beyond just using Consul Connect:

Using Helmfile to deploy Helm charts with templated chart config values, where values and branch logic is set by env vars.
Using Helmfile to patch using Kustomize merge and JSON Patch
Helm raw chart to package Kubernetes manifests as templated values
Introduction to Dgraph distributed graph database

The Challenges with Consul

You may have noticed that Consul is, dare I say, complex, beyond complex. The documentation is good, but perhaps maybe not all that well organized, with many missing things.

The underlying tool Consul is very powerful, and Consul Connect service mesh on top of this tool is quite robust and extremely flexible where you can swap out the default CA for other solutions, like Vault CA, and swap out the Envoy proxy for another solution, like NGINX or HAProxy. For ingress into the cluster, you can use Consul API Gateway, or another API Gateway or an ingress controller.

Consul Connect service mesh has some challenges or limitations (see below) when you have a service that supports multiple ports.

Complexity

I have experimented with other service meshes and I was able to get up to speed quickly: Linkerd = 1 day, Istio = 3 days, NGINX Service Mesh = 5 days, but Consul Connect service mesh took at least 11 days to get off the ground. This is by far the most complex solution available.

Unable to Update

If you need to update Consul Connect with a configuration change and use helm to update consul, the consul-server pods may not reach a healthy state. You may have to delete everything and recreate it from scratch.

Apparently there’s some way to ameliorate this by adding leave_on_terminate: true setting in the server.extraConfig (ref).

Higher Memory Footprint

Consul Connect service mesh has a higher memory footprint, so on a small cluster with e5-medium nodes (2 vCPUs, 4 GB memory), you will only be able to support a maximum of 6 side-car proxies. In order to get an application like Dgraph working, which will have 6 nodes (3 Dgraph Alpha pods and 3 Dgraph Zero pods) for high availability along with at least one client, a larger footprint with more robust Kubernetes worker nodes were required.

Requirement for Service Resource

One challenge to Consul Connect service mesh is that it configures the Envoy side-car proxy based on what you specify for a service. This added some challenges.

A pure client that is not listening on a port, still requires you to specify a service resource so that it can be added to the service mesh.
A StatefulSet that requires specifying a headless service in addition to service endpoint into the cluster will fail spectacularly if both service and headless service use the same port.

The docs explicitly note this:

Note: As of consul-k8s v0.26.0 and Consul Helm v0.32.0, having a Kubernetes service is required to run services on the Consul Service Mesh. (ref)

More Complexity with Multiport

The Kubernetes service API supports an array of ports that you can specify, but Consul Connect only supports a single port for transparent-proxy mode. This is very bizarre, because a service with multiple ports is quite common, such as an admin port vs API port, or scenarios where a service has both HTTP and gRPC interfaces.

This is also part of the Kubernetes service API specification, which Consul Connect reads to configure the Envoy proxy. So, in this sense, Kubernetes is not fully supported as far a parity with the service API.

For the multi-port scenario, the following will need to be done on the server:

all services with multiple ports will need to be broken up into separate services with only one port
need to specify consul.hashicorp.com/connect-service annotation listing each of the services supported that will be mapped into consul.
need to specify consul.hashicorp.com/connect-service-port annotation listing ports that correspond to the previous above annotation
if ACLs are enabled, a serviceaccount needs to be specified corresponding to each service specified.
if ACLs are enabled and Kubernetes 1.24+ is used, a corresponding secret for the service token needs to be created as well.

The client will need the following in order to connect to the server:

specify consul.hashicorp.com/connect-service-upstreams annotation listing the consul service and outbound port to use from localhost.
if ACLs are enabled, a serviceaccount that corresponds to the service specified for the client.

The client is now required to connect to localhost at the target outbound port, not to the service endpont DNS name, such as mysvc.myns.svc.cluster.local. This will be the only way to use the service mesh. Directly connecting to the service endpoint, e.g. mysvc.myns.svc.cluster.local, will bypass the service mesh and thus will not be protected with encryption.

Insecurity with Multiport

When transparent-proxy is enabled, members can communicate using the DNS of the service endpoint, for example: mysvc.myns.svc.cluster.local. And when you use multi-port scenario, transparent-proxy is unfortunately disabled.

Because of this situation, security through mTLS or ACLs (tokens) can be bypassed completely when multi-port services are configured. Any non-mesh member or mesh member that does not have access granted (through configuring an intention) can connect to the service endpoint, such as mysvc.myns.svc.cluster.local. The only thing ACLs offer at this point is blocking encrypted traffic through the mesh, and thus the ACL feature is pointless.

This issue can be ameliorated by configuring the service itself to only communicate through localhost, which forces it to use the service mesh, but then this poses problems, such as trying to use an ingress. Alternatively, you could use a firewall, such as a network policy. Ultimately, another non-Consul solution is needed.

Ingress Challenge with Multiport

An ingress controller is an interesting challenge to integrate to the service mesh, as annotations will be needed to put the ingress controller pods onto the service mesh. The ingress controller will route traffic to the backend service using the local DNS, such as mysvc.myns.svc.cluster.local, where the service named mysvc running in the myns namespace.

With multi-port scenario however, this will not work, because the ingress controller is now required to route to localhost for a specific outbound ports that are specified in the consul.hashicorp.com/connect-service-upstreams annotation. The normal ingress resource API does not support this setup, as it routes to Kubernetes service DNS name, not to localhost.

There may be some ingress controllers that may provide extra non-standard configurations that could support this requirement to route to localhost, but unfortunately no one at Hashicorp has even tested this common use case (ref).

No Observability with Multport

If you are using multi-port scenario, observability is not an option. Just forget you even heard of the word observability, one of the three planes that make up the service mesh solution. The Consul Connect injection process will actually cause stack traces.

https://github.com/hashicorp/consul-k8s/issues/1594

Wrapping Up

I hope this is useful in exposure to Consul Connect service mesh and can help you get started should you want to try this out. If you have services that only listen on a single port, then this certainly an interesting solution to explore.

If however, you have an application service that needs support for 2+ ports, because you know, Kubernetes supports this, I would recommend avoiding Consul Connect, as it is not functional to meet minimum requirements for a service mesh. Perhaps someday, when Hashicorp prioritizes basic functionality and usability in future version, this product can be considered.

DEV Community: Joaquin Menchaca

Architecting Kubernetes Deployments with Python

The Landscape at a Glance

1. The Most Verbose: Direct API Object Calls

Segue: Community Client Libraries

2. The Hardcoded Copy-Paste: Embedded Strings

Segue: The Cost of Tethering Configuration to Delivery Logic

3. The Gold Standard: External YAML Templates (Jinja2)

manifest.yaml.j2 (The Clean Template):

deploy.py (The Clean Script)

Beyond the Official Client

Conclusion

Practical Guidance

Further Reading

Client Libraries

Higher Level Frameworks

Articles

How AI Bridged a Tooling Gap: Extending InSpec for Kubernetes CRDs

The Problem: Trailing Ecosystems in Infrastructure-as-Code

The Mirage: A Confident Hallucination

The Twist: Engineering the Abstraction with Claude

The Real Leverage of AI in Platform Engineering

Further Reading & Resources

Minimalist EKS: The Easy Way

About eksctl

Tools

Verifying Tool Versions

Prerequisite: AWS Profile

1. Configure the Profile Setting

2. Authenticate and Activate

3. Verify Your Active Session

The Minimalist One-Command EKS Cluster

Deploying a Demo Nginx Application

Addendum: Free-Tier EKS Cluster

Addendum: Robust Minimalist Cluster

Addendum: Optional Enhancements

1. IAM OIDC Provider

2. Pod Identity Agent Addon

3. Amazon EBS CSI Driver Addon

Cleanup

Essential DevOps Tools for Ubuntu

Segue: Why Linux?

The Tools

The Installation

Repositories

Tools Installed from Repositories

Tools Installed as Binaries

Verify Installations

The Plugins

Helm Plugins

Kubernetes Plugin Engine

Kubernetes Plugins

Conclusion

Essential DevOps Tools for macOS

The Tools

Cloud Provisioning

Kubernetes

Java

Others

The Package System

The Common Core Commands

The Available Shells

The Default Shell

The Paths

The Installation

Verification

Plugins

Helm Plugins

Kubernetes Plugin Engine

Extra: The Package Manifest

Loki: Getting Started

The Server System

Using Vagrant (Virtualbox)

Installing Loki

Verify Services are running

The Client Workstation

Visualization

Ship More Logs with Promtail

Ultimate EKS Baseline Cluster: Part 2 - Storage

0. Prerequisites

`manifest.yaml.j2` (The Clean Template):

`deploy.py` (The Clean Script)