DEV Community: Anton Krylov

Why Aider

Anton Krylov — Sun, 23 Nov 2025 20:37:34 +0000

The market for AI coding assistants has split into two clearly defined camps.

Camp #1 – Full-featured graphical IDEs

Cursor, Windsurf, Zed + Cursor mode, VS Code with Continue + Copilot + many extensions, IntelliJ Ultimate with AI Assistant, etc.

Polished UI, inline completions, chat panels, debugging integration, project-wide indexing.

Typical costs: 8–40 s startup, 4–8 GB RAM, mandatory indexing of the whole repo, poor or no support for direct work over SSH on production/legacy servers.

Camp #2 – Terminal-first tools

vim, neovim, helix, kakoune, tmux… and aider.

Start in 200–400 ms, use 50–150 MB RAM, require only a terminal and git, work perfectly over SSH with no indexing step.

aider is a command-line tool that sends selected files + conversation to an LLM and applies the returned edits directly under git control. It is widely used by senior engineers, DevOps, SRE, and security teams for fast, targeted changes in large or old repositories where launching a heavy IDE is impractical.

Configuration

aider is configured via ~/.aider.conf.yml (or .aider.conf.yml in the repository) and command-line flags.

Common options include:

choice of main and weak models
read: list to permanently attach markdown files with guidelines, conventions, security rules, etc.
toggles for auto-commits, pretty diffs, vim mode, prompt caching, etc.

One can use multiple profile YAML files placed in ~/.aider/profiles/ and switch between them with a little script that copies the selected profile to the active config file.

Here's a full example of such a script (/usr/local/bin/aider-switch):

#!/usr/bin/env bash
# aider-switch — switch profiles with a single command
set -euo pipefail
AIDER_DIR="$HOME/.aider"
PROFILES_DIR="${AIDER_DIR}/profiles"
CURRENT_LINK="${AIDER_DIR}/current"
CONFIG_FILE="${AIDER_DIR}/aider.conf.yml"

die() { echo "Error: $*" >&2; exit 1; }

[[ -d "$AIDER_DIR" ]] || die ".aider/ not found in $HOME"
[[ -d "$PROFILES_DIR" ]] || die ".aider/profiles/ missing in $HOME"

case "${1:-}" in
  list|"")
    echo "Available profiles:"
    ls -1 "$PROFILES_DIR"/*.yaml 2>/dev/null | xargs -n1 basename | sed 's/\.yaml$//' | sed 's/^/  /'
    [[ -f "$CONFIG_FILE" ]] && echo -e "\nCurrent → $(yq e '.model // "unknown"' "$CONFIG_FILE" 2>/dev/null || echo "unknown")"
    exit 0
    ;;
  *)
    PROFILE="$1"
    SRC="$PROFILES_DIR/$PROFILE.yaml"
    [[ -f "$SRC" ]] || { echo "Profile '$PROFILE' not found"; exit 1; }
    cp -f "$SRC" "$CONFIG_FILE"
    ln -sf "$(basename "$SRC")" "$CURRENT_LINK" 2>/dev/null || true
    echo "Switched to profile → $PROFILE"
    yq e '.model // "unknown"' "$CONFIG_FILE" 2>/dev/null | sed 's/^/ model: /'
    ;;
esac

Usage:

aider-switch list
aider-switch security-review
aider-switch kubernetes-expert && aider

You keep a handful of YAML files in ~/.aider/profiles/ and a few markdown rule sheets nearby. One profile for backend work loads grok-4, turns on vim mode, and permanently attaches backend.md plus conventions-security.md. Another for security reviews swaps to claude-3.5-sonnet, cranks caution up, and locks onto security-review.md alone. DevOps profile picks a cheaper weak model and reads only kubernetes.md and ansible-conventions.md. Switch with aider-switch backend or aider-switch security-review and the model instantly forgets yesterday’s tone and remembers exactly who it is supposed to be today. Ten lines of bash give you twenty different personalities for the same binary. That is all there is to it.

Prompting is 95 % of success with aider. aider never tries to be smarter than you — it faithfully executes exactly what’s written in the profiles and the current request. No surprise refactoring, no “I thought this would be better.”

The markdown files loaded via read: act as permanent code review from your lead architect and security lead on every edit. The stricter and more concrete the rules, the safer and more predictable the output — even on the most powerful models.

Bad/empty profile + vague prompt = production chaos Good profile = katana instead of chainsaw

That’s why the real work with aider isn’t picking a model; it’s writing and maintaining evergreen profiles.

Example ~/.aider/profiles/backend.md:

# Backend Instructions
- Define clear APIs and contracts; version endpoints when needed.
- Validate and sanitize all inputs; enforce types and limits.
- Implement authn/authz consistently; least privilege by default.
- Handle errors with structured responses; avoid leaking internals.
- Add observability: logs, metrics, and traces for critical paths.
- Manage migrations carefully; ensure idempotent operations and rollbacks.
- Use timeouts/retries/circuit breakers for external calls.
- Protect secrets and credentials; rotate and scope access.

These files are loaded into context on every run and survive /clear. Write once — never repeat again.

Typical usage examples

aider src/api/users/*.py
# Migrate to Pydantic v2, follow the attached backend guidelines

aider --message-file security.md src/auth/*.py
# Look for SSRF, open redirects and credential leaks

aider deploy/values-prod.yaml
# Increase replicas to 12 and add the new nodeSelector

Security-sensitive environments

In security-sensitive environments (banks, healthcare, defense, government), running aider directly on a developer’s laptop is often forbidden by compliance. A single careless prompt could leak secrets to logs, exfiltrate context, or generate malicious code.
The gold standard fix: run aider inside an isolated Firecracker microVM (or gVisor/Kata). The microVM boots in ~120–150 ms, has its own kernel, zero network access, and only the current git repository is mounted — nothing else. Host keys, credentials, other projects, Docker sockets, and cloud access remain physically unreachable by the LLM process.

https://github.com/avkcode/firecracker-sandbox let you spin up the sandbox with one command while keeping your familiar vim/tmux workflow intact. The developer feels no difference, but the model is completely contained.

Summary

Heavy graphical AI IDEs and lightweight terminal tools such as aider address different workflows.

The graphical tools are preferred for long, exploratory development sessions on a local workstation.

Terminal tools are chosen when minimal latency, low resource usage, seamless SSH access, and strict control over context and edits take priority.

Most engineers use both categories, switching depending on the specific task.

Redroid

Anton Krylov — Wed, 11 Jun 2025 09:47:52 +0000

What is ReDroid?

ReDroid is a lightweight alternative to the standard Android emulator that runs as a Docker container. ReDroid provides a full Android system in a container, significantly reducing startup time and resource consumption compared to traditional emulators.

ReDroid uses the host system's Linux kernel and is based on the anbox module project, enabling Android to run without CPU virtualization. This makes it an ideal solution for automated Android app testing in CI/CD environments where speed and efficiency are critical.

Comparison with Android Studio Emulator:

ReDroid is perfect for automated UI testing in CI/CD, especially when speed and resource efficiency are important. The standard Android Studio emulator is better suited for local development, debugging, and testing requiring full device emulation.

Manual Testing

Start the ReDroid container (Docker) and wait about a minute for it to load:

docker run -d --name redroid --privileged -p 5555:5555 redroid/redroid:15.0.0_64only-latest
sleep 30

Connect to the running container via ADB:

adb connect localhost:5555
adb devices

Install both APK files on ReDroid:

adb -s localhost:5555 install ./app/build/outputs/apk/debug/app-debug.apk
adb -s localhost:5555 install ./app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk

Run your Espresso/UI tests with ADB:

adb -s localhost:5555 shell am instrument -w your.package.name.test/androidx.test.runner.AndroidJUnitRunner

Export logs after testing:

adb -s localhost:5555 logcat -d > android-log.txt

After completing the tests, remove the container:

docker stop redroid && docker rm redroid

Preparing an Android Project for UI Testing

Step 1: Configuring Dependencies in app/build.gradle

gradle:

android {
    defaultConfig {
        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
    }
}

dependencies {
    // Core components for UI testing
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.5.1'
    androidTestImplementation 'androidx.test:runner:1.5.2'
    androidTestImplementation 'androidx.test:rules:1.5.0'

    // Additional Espresso libraries for advanced testing
    androidTestImplementation 'androidx.test.espresso:espresso-contrib:3.5.1'
    androidTestImplementation 'androidx.test.espresso:espresso-intents:3.5.1'

    // JUnit for test environment integration
    androidTestImplementation 'androidx.test.ext:junit:1.1.5'
}

Step 2: Creating an Example UI Test with Espresso

Create the file app/src/androidTest/java/com/example/app/MainActivityTest.java:

java:

package com.example.app;

import androidx.test.espresso.matcher.ViewMatchers;
import androidx.test.ext.junit.rules.ActivityScenarioRule;
import androidx.test.ext.junit.runners.AndroidJUnit4;
import androidx.test.filters.LargeTest;

import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;

import static androidx.test.espresso.Espresso.onView;
import static androidx.test.espresso.action.ViewActions.click;
import static androidx.test.espresso.assertion.ViewAssertions.matches;
import static androidx.test.espresso.matcher.ViewMatchers.isDisplayed;
import static androidx.test.espresso.matcher.ViewMatchers.withId;
import static androidx.test.espresso.matcher.ViewMatchers.withText;

@RunWith(AndroidJUnit4.class)
@LargeTest
public class MainActivityTest {

    @Rule
    public ActivityScenarioRule<MainActivity> activityRule =
        new ActivityScenarioRule<>(MainActivity.class);

    @Test
    public void welcomeTextIsDisplayed() {
        // Check that the welcome text field is displayed
        onView(withId(R.id.welcome_text))
            .check(matches(isDisplayed()))
            .check(matches(withText("Hello World!")));
    }

    @Test
    public void buttonClickChangesText() {
        // Click the button
        onView(withId(R.id.action_button))
            .perform(click());

        // Check that the text has changed
        onView(withId(R.id.welcome_text))
            .check(matches(withText("Button Clicked!")));
    }
}

Configuring Jenkins for UI Test Automation with ReDroid

Step 1: Installing Required Jenkins Plugins

In the Jenkins interface, go to "Configure Jenkins" > "Manage Plugins" and install:

Docker Pipeline
Gradle Plugin
HTML Publisher Plugin
JUnit Plugin
Allure Jenkins Plugin (optional for enhanced reporting)
Step 2: Creating a Jenkinsfile for the Pipeline

Create a Jenkinsfile in the project root:

groovy:

pipeline {
    agent any

    environment {
        // Environment variables
        ANDROID_HOME = '/opt/android-sdk'
        REDROID_IMAGE = 'redroid/redroid:15.0.0_64only-latest'
        REDROID_CONTAINER = 'android-ui-test-container'
        ADB_PORT = '5555'
        TEST_RESULTS_DIR = 'app/build/outputs/androidTest-results'
        TEST_REPORT_DIR = 'app/build/reports/androidTests'
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Build APKs') {
            steps {
                sh './gradlew clean assembleDebug assembleDebugAndroidTest'
            }
        }

        stage('Start ReDroid Container') {
            steps {
                script {
                    // Stop and remove the container if it exists
                    sh "docker stop ${REDROID_CONTAINER} || true"
                    sh "docker rm ${REDROID_CONTAINER} || true"

                    // Run ReDroid in Docker
                    sh """
                        docker run -d --name ${REDROID_CONTAINER} \
                        --privileged \
                        -p ${ADB_PORT}:${ADB_PORT} \
                        -e REDROID_PROP_ro.debuggable=1 \
                        -e REDROID_PROP_service.adb.tcp.port=${ADB_PORT} \
                        ${REDROID_IMAGE}
                    """

                    // Wait for ReDroid to start
                    sh "sleep 30"
                }
            }
        }

        stage('Connect to ReDroid and Install APKs') {
            steps {
                script {
                    // Connect to the device via ADB
                    sh "adb connect localhost:${ADB_PORT}"
                    sh "adb devices"

                    // Wait for the system to fully load
                    sh "adb wait-for-device"

                    // Install the app APK and tests
                    sh "adb -s localhost:${ADB_PORT} install -r app/build/outputs/apk/debug/app-debug.apk"
                    sh "adb -s localhost:${ADB_PORT} install -r app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk"
                }
            }
        }

        stage('Run UI Tests') {
            steps {
                script {
                    // Create directories for results
                    sh "mkdir -p ${TEST_RESULTS_DIR}"

                    // Get the app package name
                    def packageName = sh(
                        script: "aapt dump badging app/build/outputs/apk/debug/app-debug.apk | grep package | awk '{print \$2}' | sed s/name=//g | sed s/\\'//g",
                        returnStdout: true
                    ).trim()

                    // Run UI tests
                    sh """
                        adb -s localhost:${ADB_PORT} shell am instrument -w \
                        -e debug false \
                        -e junit.output.format=xml \
                        -e additionalTestOutputDir=/sdcard/test-results \
                        ${packageName}.test/androidx.test.runner.AndroidJUnitRunner
                    """

                    // Copy results from the device
                    sh "adb -s localhost:${ADB_PORT} pull /sdcard/test-results ${TEST_RESULTS_DIR}"

                    // Collect logs
                    sh "adb -s localhost:${ADB_PORT} logcat -d > ${TEST_RESULTS_DIR}/device-log.txt"

                    // Take a screenshot for diagnostics (optional)
                    sh "adb -s localhost:${ADB_PORT} shell screencap -p /sdcard/screen.png"
                    sh "adb -s localhost:${ADB_PORT} pull /sdcard/screen.png ${TEST_RESULTS_DIR}/"
                }
            }
        }
    }

    post {
        always {
            // Publish JUnit results
            junit "${TEST_RESULTS_DIR}/**/*.xml"

            // Publish test reports
            publishHTML([
                allowMissing: true,
                alwaysLinkToLastBuild: true,
                keepAll: true,
                reportDir: "${TEST_REPORT_DIR}/connected",
                reportName: 'Espresso Test Report',
                reportFiles: 'index.html'
            ])

            // Stop and remove the container
            script {
                sh "docker stop ${REDROID_CONTAINER} || true"
                sh "docker rm ${REDROID_CONTAINER} || true"
            }

            // Clean up ADB
            sh "adb disconnect localhost:${ADB_PORT} || true"

            // Archive artifacts
            archiveArtifacts artifacts: "${TEST_RESULTS_DIR}/**/*", allowEmptyArchive: true
        }
    }
}

Scrcpy

Scrcpy is a lightweight, free, open-source utility for displaying and controlling Android device screens on a computer. It works over USB (ADB) or Wi-Fi without requiring any mobile apps on the device.

Usage Examples:

Connect and start (device connected via USB):

scrcpy

Connect via Wi-Fi (knowing the device's IP address):

adb connect 192.168.1.5:5555
scrcpy -s 192.168.1.5:5555

Change video resolution:

scrcpy -m 1024

Record device video:

scrcpy --record file.mp4

Control the device without displaying the screen (keyboard and mouse input only):

scrcpy -N

If you need visual monitoring of test execution, you can integrate Scrcpy into the pipeline:

stage('Visual Debug with Scrcpy') {
    when {
        expression { return params.ENABLE_VISUAL_DEBUG }
    }
    steps {
        script {
            // Install Scrcpy if not already installed
            sh '''
                if ! command -v scrcpy &> /dev/null; then
                    apt-get update && apt-get install -y scrcpy
                fi
            '''

            // Start Scrcpy to record test execution
            sh "mkdir -p ${TEST_RESULTS_DIR}/recordings"

            for (int i = 0; i < DEVICE_COUNT; i++) {
                def adbPort = BASE_ADB_PORT + i
                sh """
                    nohup scrcpy --no-display --record=${TEST_RESULTS_DIR}/recordings/device-${i}-recording.mp4 \
                    --serial=localhost:${adbPort} &
                """
            }

            // Limit recording duration
            sh "sleep 60"  // Record the first minute of testing
            sh "pkill scrcpy || true"
        }
    }

Kubernetes

Why Use ReDroid in Kubernetes

ReDroid in Kubernetes simplifies Android app testing. Here are the key benefits:

Parallel Tests — Run multiple virtual Android devices simultaneously, saving testing time.

Lower Resource Costs — ReDroid is lightweight, and Kubernetes efficiently distributes the load across servers.

Automatic Management — The system automatically starts, restarts, and stops Android containers without manual intervention.

Easy CI/CD Integration — Seamlessly integrates with existing Jenkins or GitLab CI pipelines.

Test Isolation — Each test environment operates independently without interfering with others.

Centralized Management — Everything is controlled through a single interface.

Budget Savings — Virtual devices replace expensive physical phones.

Consistent Conditions — All tests run in the same environment, improving result reliability.

Hardware Acceleration Access — Can be configured to use GPU for better performance.

This solution is ideal for teams needing to regularly test Android apps in different configurations quickly and predictably.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redroid-farm
  namespace: android-testing
spec:
  serviceName: "redroid"
  replicas: 3  # Number of parallel device instances
  selector:
    matchLabels:
      app: redroid-device
  template:
    metadata:
      labels:
        app: redroid-device
    spec:
      securityContext:
        runAsUser: 0  # Run as root
      containers:
      - name: redroid
        image: redroid/redroid:15.0.0_64only-latest
        securityContext:
          privileged: true  # Required for redroid
        ports:
        - containerPort: 5555
          name: adb
        env:
        - name: REDROID_PROP_ro.debuggable
          value: "1"
        - name: REDROID_PROP_service.adb.tcp.port
          value: "5555"
        - name: ANDROID_ARCH
          value: "x86_64"
        resources:
          requests:
            cpu: 2
            memory: 3Gi
          limits:
            cpu: 4
            memory: 6Gi
        volumeMounts:
        - name: dri
          mountPath: /dev/dri
        - name: kvm
          mountPath: /dev/kvm
        lifecycle:
          preStop:
            exec:
              command: ["/system/bin/reboot", "-p"]
      volumes:
      - name: dri
        hostPath:
          path: /dev/dri
      - name: kvm
        hostPath:
          path: /dev/kvm
      nodeSelector:
        hardware-acceleration: "true"  # Select nodes with GPU/hardware acceleration

Wireshark and ReDroid: Capturing and Analyzing Android App Network Traffic

Wireshark combined with ReDroid provides powerful tools for testing and debugging Android app network interactions. This combination is particularly useful for analyzing API requests, checking connection security, and debugging network issues.

Key Features:

Real Traffic Monitoring — Wireshark lets you see all network packets exchanged between the Android app in ReDroid and external services.
Easy Setup — Since ReDroid runs as a Docker container, its network traffic is easily captured through Docker networking without additional tools on the device itself.
Secure Traffic Analysis — With proper setup, you can even inspect HTTPS traffic (using tools like mitmproxy with ReDroid).
Performance Issue Detection — Easily identify slow requests and app performance delays from network data.
Automated API Testing — Record and replay network interaction scenarios for testing.

Network Setup for Testing:

Run ReDroid with special network parameters:

docker run --name redroid-test --network host redroid/redroid:11.0.0-latest

Configure traffic capture in Wireshark, filtering for the specific ReDroid container traffic:

ip.addr == [container IP address]

For HTTPS traffic, configure a proxy in ReDroid via ADB:

adb connect localhost:5555
adb shell settings put global http_proxy "localhost:8080"

Start mitmproxy for HTTPS decryption:

mitmproxy --listen-port 8080

This solution helps developers and testers better understand how their Android apps interact with the network, identify security issues, and optimize network interactions without needing physical devices or complex emulator setups.

Terraform and Ansible is not enough

Anton Krylov — Sat, 17 May 2025 16:32:30 +0000

Terraform and Ansible is not enough

Tabel of contents

Preface
Ansible without/with Temporal
Simple Workflow
Long-Running Workflows with Checkpointing
Dynamic Parallel Execution
Human-in-the-Loop Approvals
Cross-Cloud Orchestration
Event-Driven Ansible (EDA) on Steroids
Stateful Workflows with Recovery
Time Travel Debugging
Cross-Tool Chaining
Dynamic Ansible Inventory
GitLab Integration
Terraform
Database Migrations

Preface

When it comes to orchestrating complex workflows in hybrid cloud environments, Temporal stands out as one of the best open-source tools available. It doesn’t try to replace tools like Terraform or Ansible but instead complements them by adding a layer of reliability, scalability, and flexibility.

For example, while Terraform excels at provisioning infrastructure and Ansible handles configuration management, Temporal ties everything together. It ensures that workflows can recover from failures, adapt to runtime conditions, and integrate seamlessly with external systems like monitoring tools, CMDBs, or approval processes. And because it’s open source, it avoids vendor lock-in while supporting virtually any technology stack.

The real value of workflow orchestrator lies in its ability to handle long-running, dynamic workflows without skipping a beat. Whether it’s waiting for a VM to pass health checks, triggering a Kubernetes deployment, or updating a CMDB after provisioning, Temporal keeps things running smoothly. It’s not about reinventing the wheel—it’s about making the tools you already use work better, together.

Another advantage is how Temporal handles long-running processes common in Day 2 operations. For instance, if you’re migrating workloads between clouds or performing a rolling update across a Kubernetes cluster, Temporal keeps track of progress and ensures the workflow resumes from where it left off—even if there’s a failure or interruption. This makes it ideal for tasks like database upgrades, application rollbacks, or compliance audits, where consistency and reliability are critical.

In short, Temporal helps IT teams move faster and more efficiently without sacrificing control or governance, which is exactly what modern enterprises need in today’s fast-changing hybrid cloud world.

Ansible with Temporal

Temporal addresses challenges in automating workflows by providing a robust framework for managing complex, long-running processes that require reliability, scalability, and fault tolerance. Without Temporal, running Ansible playbooks or similar automation tasks can face issues like failed executions due to transient errors, lack of visibility into ongoing processes, inability to resume workflows after system crashes, and difficulty in coordinating multiple dependent tasks. Temporal ensures that workflows can recover gracefully from failures, maintain state across retries, and allow dynamic adjustments based on runtime conditions. It also enables seamless integration of different tools and systems within a single workflow, making it easier to orchestrate multi-step operations that involve not just Ansible but potentially other infrastructure or application management tools. By handling the complexities of task coordination, retries, and state management, Temporal allows developers to focus on defining the logic of their workflows rather than worrying about the underlying execution mechanics.

graph TD
    %% Custom Styling
    classDef temporal fill:#4CAF50,stroke:#333,stroke-width:2px,color:#fff;
    classDef ansible fill:#FF9800,stroke:#333,stroke-width:2px,color:#fff;
    classDef problem fill:#F44336,stroke:#333,stroke-width:2px,color:#fff;
    classDef solution fill:#2196F3,stroke:#333,stroke-width:2px,color:#fff;

    %% Nodes
    A[Challenges Without Temporal]:::problem --> B["Failed Executions<br>(Transient Errors)"];
    A --> C["Lack of Visibility<br>(No Monitoring)"];
    A --> D["Inability to Resume<br>(Crash Recovery)"];
    A --> E["Coordination Issues<br>(Dependent Tasks)"];

    F[Temporal Framework]:::temporal --> G["Retry Mechanisms<br>(Fault Tolerance)"];
    F --> H["State Management<br>(Persist State Across Failures)"];
    F --> I["Dynamic Adjustments<br>(Runtime Conditions)"];
    F --> J["Multi-Tool Integration<br>(Ansible + Others)"];

    K[Benefits of Temporal]:::solution --> L["Graceful Recovery<br>(From Failures)"];
    K --> M["Maintain State<br>(Across Retries)"];
    K --> N["Orchestrate Multi-Step<br>(Complex Operations)"];
    K --> O["Focus on Logic<br>(Not Execution Mechanics)"];

    %% Flow
    B -->|Solved By| F
    C -->|Solved By| F
    D -->|Solved By| F
    E -->|Solved By| F

    F -->|Enables| K

    %% Subgraph for Grouping Temporal Features
    subgraph TemporalFeatures
        direction TB
        G[Retry Mechanisms]
        H[State Management]
        I[Dynamic Adjustments]
        J[Multi-Tool Integration]
    end

Simple workflow

This Python script shows how to use Temporal to run an Ansible playbook, like one that updates system packages on a server. It sets up a workflow and activity to execute the playbook in a reliable way, with features like retries, live output streaming, and clean shutdowns.

The execute_ansible_playbook activity runs the ansible-playbook command using asyncio to capture and display the output as it happens. You can pass the playbook file and, optionally, an inventory file. If something goes wrong and the playbook fails, the script raises an error, and Temporal automatically retries the operation up to three times before stopping.

The AnsibleWorkflow defines the workflow logic, which basically calls the activity to run the playbook. It’s set up with timeouts and retry policies to handle cases where the playbook takes a long time or runs into temporary issues.

When you run the script, it connects to a Temporal server, starts a worker to listen for tasks, and immediately kicks off the workflow to run the playbook you specify. For example, you could point it to a playbook like update_packages.yml, which might use Ansible modules like apt or yum to update packages on your servers. The script ensures the playbook runs smoothly, streams its output in real-time, and handles interruptions or errors without losing track of what’s happening.

This approach is great for automating tasks like updating packages across servers, where reliability and visibility are key. By using Temporal, you get built-in retries, fault tolerance, and the ability to monitor and debug the process easily if something doesn’t go as planned.

import asyncio
import uuid
import sys
import os
import signal
import argparse
from datetime import timedelta
from temporalio import workflow, activity
from temporalio.client import Client
from temporalio.worker import Worker
from temporalio.common import RetryPolicy

# --- Activity Definition ---
@activity.defn
async def execute_ansible_playbook(playbook_path: str, inventory_path: str = None) -> str:
    """Execute ansible-playbook with live output."""
    cmd = ["ansible-playbook", playbook_path]
    if inventory_path:
        cmd.extend(["-i", inventory_path])

    proc = await asyncio.create_subprocess_exec(
        *cmd,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE
    )

    output = []
    while True:
        stdout_chunk = await proc.stdout.read(1024)
        stderr_chunk = await proc.stderr.read(1024)

        if not stdout_chunk and not stderr_chunk:
            break

        if stdout_chunk:
            sys.stdout.write(stdout_chunk.decode())
            sys.stdout.flush()
            output.append(stdout_chunk.decode())

        if stderr_chunk:
            sys.stderr.write(stderr_chunk.decode())
            sys.stderr.flush()
            output.append(f"ERROR: {stderr_chunk.decode()}")

    exit_code = await proc.wait()
    if exit_code != 0:
        raise RuntimeError(f"Playbook failed with exit code {exit_code}")

    return "".join(output)

# --- Workflow Definition ---
@workflow.defn
class AnsibleWorkflow:
    @workflow.run
    async def run(self, params: dict) -> str:
        return await workflow.execute_activity(
            execute_ansible_playbook,
            args=[params["playbook_path"], params.get("inventory_path")],
            start_to_close_timeout=timedelta(minutes=15),
            retry_policy=RetryPolicy(
                initial_interval=timedelta(seconds=30),
                maximum_interval=timedelta(minutes=2),
                maximum_attempts=3
            )
        )

async def run_workflow_and_worker(playbook_path: str, inventory_path: str = None):
    """Start worker and execute workflow automatically"""
    shutdown_event = asyncio.Event()
    client = None
    worker = None

    def signal_handler():
        shutdown_event.set()

    loop = asyncio.get_running_loop()
    loop.add_signal_handler(signal.SIGINT, signal_handler)
    loop.add_signal_handler(signal.SIGTERM, signal_handler)

    try:
        # Connect to Temporal
        client = await Client.connect("localhost:7233")

        # Start worker
        worker = Worker(
            client,
            task_queue="ansible-queue",
            workflows=[AnsibleWorkflow],
            activities=[execute_ansible_playbook],
            graceful_shutdown_timeout=timedelta(seconds=5)
        )

        # Run worker in background
        worker_task = asyncio.create_task(worker.run())

        # Execute workflow
        print(f"Executing playbook: {playbook_path}")
        result = await client.execute_workflow(
            AnsibleWorkflow.run,
            args=[{"playbook_path": playbook_path, "inventory_path": inventory_path}],
            id=f"ansible-{uuid.uuid4()}",
            task_queue="ansible-queue",
            execution_timeout=timedelta(minutes=20)
        )

        print("\nPlaybook execution result:")
        print(result)

    except Exception as e:
        print(f"Error: {str(e)}")
        raise
    finally:
        # Clean shutdown
        shutdown_event.set()
        if worker:
            await worker.shutdown()

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--playbook", required=True, help="Path to playbook file")
    parser.add_argument("--inventory", help="Path to inventory file")
    args = parser.parse_args()

    # Validate paths
    if not os.path.exists(args.playbook):
        print(f"Error: Playbook not found at {args.playbook}")
        sys.exit(1)
    if args.inventory and not os.path.exists(args.inventory):
        print(f"Error: Inventory file not found at {args.inventory}")
        sys.exit(1)

    try:
        asyncio.run(run_workflow_and_worker(args.playbook, args.inventory))
    except KeyboardInterrupt:
        print("\nProcess stopped by user")
        sys.exit(0)
    except Exception as e:
        print(f"\nError: {str(e)}")
        sys.exit(1)

This command lists all workflows of type AnsibleWorkflow that are currently tracked by Temporal:

tctl workflow list --query "WorkflowType='AnsibleWorkflow'"

This command starts a new instance of the AnsibleWorkflow to execute the apt_update.yml playbook:

tctl workflow run \
    --taskqueue ansible-queue \
    --workflow_type AnsibleWorkflow \
    --input '"apt_update.yml"'

--inventory is optional in this case.

This command retrieves detailed information about a specific workflow instance identified by its workflow:

tctl workflow show -w ansible-1dcbe2e1-8b10-4616-8e11-948cb81d83b1

Temporal UI simplifies workflow management by providing an intuitive interface to check statuses, inspect details, and take actions like canceling workflows when necessary.

This GitLab pipeline job, ansible_workflows, automates the execution of an Ansible playbook like apt_update.yml to update system packages. It runs in the deploy stage using a shell executor and triggers a Python script (ansible.py) that orchestrates the playbook via Temporal.

The script uses Temporal to run the playbook reliably, with features like retries and live output streaming. If the playbook fails, the pipeline reflects the error for easy debugging. This setup integrates infrastructure updates into your CI/CD process, ensuring reliable and automated deployments.

ansible_workflows:
  stage: deploy
  tags:
    - shell
  script:
    - python3 ansible.py --playbook apt_update.yml

Long-Running Workflows with Checkpointing

Problem: Ansible playbooks fail on long-running tasks (e.g., cloud provisioning, multi-hour deployments).
Solution:
Use Temporal workflows to automatically resume from failures.
Checkpoint progress (e.g., "50% of VMs deployed") even if the process crashes.

@workflow.defn
class AnsibleWorkflow:
    async def run(self, playbook: str):
        while not workflow.is_replaying():
            result = await workflow.execute_activity(
                run_ansible_playbook,
                args=[playbook],
                start_to_close_timeout=timedelta(hours=6),
                heartbeat_timeout=timedelta(minutes=5),
            )
            if result.failed:
                await asyncio.sleep(300)  # Retry after 5 minutes

sequenceDiagram
    participant User
    participant Temporal
    participant Ansible
    User->>Temporal: Start Workflow
    Temporal->>Ansible: Run Playbook
    Ansible-->>Temporal: Progress (50% done)
    Note over Temporal: Crash occurs
    Temporal->>Temporal: Resume from checkpoint
    Temporal->>Ansible: Retry failed tasks
    Ansible-->>Temporal: Task completed
    Temporal-->>User: Workflow completed

Dynamic Parallel Execution

Problem: Ansible’s async is limited—hard to manage 1000s of nodes dynamically.
Solution:
Fan-out/fan-in workflows (e.g., deploy 500 VMs in parallel, then aggregate results).

async def deploy_vms():
    vm_list = await get_vm_list_from_api()
    results = await asyncio.gather(
        *[run_ansible_playbook(f"deploy_vm.yml --extra-vars 'vm_id={vm.id}'") for vm in vm_list]
    )
    return sum(results)

graph TD
    A[Start Workflow] --> B[Fetch VM List]
    B --> C[Fan-Out: Deploy VMs in Parallel]
    C --> D[Deploy VM 1]
    C --> E[Deploy VM 2]
    C --> F[Deploy VM 3]
    D --> G[Aggregate Results]
    E --> G
    F --> G
    G --> H[Workflow Completed]

Human-in-the-Loop Approvals

Problem: Ansible Tower requires manual static approvals.
Solution:
Dynamic pauses (e.g., "Wait for Slack approval before deleting prod DB").

@workflow.defn
class ProdDeploymentWorkflow:
    async def run(self):
        await run_ansible_playbook("deploy_staging.yml")
        if workflow.is_replaying():
            await workflow.wait_for_signal("prod_approval")
        else:
            await send_slack_approval_request()
            await workflow.wait_for_signal("prod_approval")  # Blocks until approved
        await run_ansible_playbook("deploy_prod.yml")

sequenceDiagram
    participant Workflow
    participant Slack
    participant User
    Workflow->>Slack: Send Approval Request
    Slack->>User: Notify for Approval
    User->>Slack: Approve
    Slack->>Workflow: Signal Approval
    Workflow->>Workflow: Proceed to Next Step

Cross-Cloud Orchestration

Problem: Ansible alone can’t coordinate AWS + Azure + GCP workflows.
Solution:
Temporal workflows orchestrate multi-cloud playbooks:

async def migrate_to_aws():
    await run_ansible_playbook("azure_shutdown.yml")
    await run_ansible_playbook("aws_provision.yml")
    if await check_aws_health():
        await run_ansible_playbook("cutover_dns.yml")

graph TD
    A[Start Workflow] --> B[Shutdown Azure Resources]
    B --> C[Provision AWS Resources]
    C --> D[Check AWS Health]
    D --> E[Update DNS]
    E --> F[Workflow Completed]

Event-Driven Ansible (EDA) on Steroids

Problem: Ansible EDA reacts to simple events (e.g., webhooks).
Solution:
Complex event chains (e.g., "If CPU > 90% for 5min, scale + notify PagerDuty + ticket").

async def auto_scale_workflow():
    while True:
        metrics = await fetch_cloud_metrics()
        if metrics.cpu > 90:
            await run_ansible_playbook("scale_out.yml")
            await page_team()
            await workflow.sleep(timedelta(minutes=5))  # Cooldown

sequenceDiagram
    participant Monitor
    participant Workflow
    participant Ansible
    participant PagerDuty
    Monitor->>Workflow: CPU > 90% for 5min
    Workflow->>Ansible: Scale Out
    Workflow->>PagerDuty: Notify Team
    Workflow->>Workflow: Cooldown for 5min

Stateful Workflows with Recovery

Problem: Ansible has no memory of past runs.
Solution:
Temporal remembers state (e.g., "Retry only failed nodes after outage").

async def patch_workflow():
    hosts = await get_hosts()
    for host in hosts:
        try:
            await run_ansible_playbook(f"patch.yml -l {host}")
        except ActivityError:
            await log_failed_host(host)
    if workflow.is_replaying():
        await retry_failed_hosts()  # Only retries what crashed

graph TD
    A[Start Workflow] --> B[Get Host List]
    B --> C[Run Playbook on Host 1]
    B --> D[Run Playbook on Host 2]
    B --> E[Run Playbook on Host 3]
    C --> F[Log Failed Host]
    D --> F
    E --> F
    F --> G[Retry Failed Hosts]
    G --> H[Workflow Completed]

Time Travel Debugging

Problem: Debugging Ansible failures is painful.
Solution:
Replay workflows exactly (e.g., "See why playbook failed 3 days ago").

# Temporal UI shows full history + inputs/outputs for every step.

sequenceDiagram
    participant User
    participant Temporal
    User->>Temporal: Replay Workflow
    Temporal->>Temporal: Load History
    Temporal->>Temporal: Re-execute Steps
    Temporal-->>User: Show Failure Details

Cross-Tool Chaining

Problem: Ansible can’t seamlessly call Terraform + Kubernetes.
Solution:
Mix tools in one workflow:

async def deploy_full_stack():
    await run_terraform("apply")
    await run_kubectl("apply -f k8s/")
    await run_ansible_playbook("configure_ingress.yml")

graph TD
    A[Start Workflow] --> B[Run Terraform Apply]
    B --> C[Run Kubectl Apply]
    C --> D[Run Ansible Playbook]
    D --> E[Workflow Completed]

Dynamic Ansible inventory

Here’s how to create a dynamic Ansible inventory using a Temporal workflow that fetches host data from a CMDB via REST API, ensuring real-time, fault-tolerant infrastructure management:

Temporal Workflow → Fetches CMDB Data (REST API) → Generates Dynamic Inventory → Runs Ansible

def fetch_cmdb_hosts(cmdb_api_url: str, token: str) -> dict:
    headers = {"Authorization": f"Bearer {token}"}
    response = requests.get(f"{cmdb_api_url}/hosts", headers=headers)
    response.raise_for_status()
    return response.json()

def generate_inventory(cmdb_data: dict) -> dict:
    inventory = {
        "_meta": {"hostvars": {}},
        "all": {"hosts": []},
        "web": {"hosts": []},
        "db": {"hosts": []}
    }

    for host in cmdb_data["hosts"]:
        inventory["all"]["hosts"].append(host["name"])
        inventory["_meta"]["hostvars"][host["name"]] = {
            "ansible_host": host["ip"],
            "ansible_user": host["user"],
            "ansible_become": True
        }
        if host["role"] == "web":
            inventory["web"]["hosts"].append(host["name"])
        elif host["role"] == "db":
            inventory["db"]["hosts"].append(host["name"])

    return inventory

graph TD
    A[Start Workflow] --> B[Fetch CMDB Data]
    B --> C[Generate Dynamic Inventory]
    C --> D[Run Ansible Playbook]
    D --> E[Workflow Completed]

GitLab

To integrate the Temporal-based Ansible workflow with a GitLab webhook , you can configure GitLab to trigger the workflow whenever specific events occur (e.g., a push to a branch, a merge request, or a tag creation).

sequenceDiagram
    participant GitLab
    participant WebhookHandler
    participant Temporal
    participant Ansible
    GitLab->>WebhookHandler: Send webhook payload (Push to main)
    WebhookHandler->>WebhookHandler: Validate payload
    WebhookHandler->>Temporal: Trigger workflow
    Temporal->>Ansible: Execute playbook (apt_update.yml)
    Ansible-->>Temporal: Playbook completed
    Temporal-->>WebhookHandler: Workflow completed
    WebhookHandler-->>GitLab: Acknowledge webhook

Terraform

Using Terraform and Temporal together offers a powerful combination for infrastructure automation, addressing gaps that arise when using either tool in isolation. Terraform excels at declarative infrastructure provisioning but requires custom providers for new or niche services, which can be time-consuming to develop and maintain. On the other hand, Temporal provides a robust orchestration layer for automating workflows, including those involving APIs, without the need for custom providers. For example, you can use Temporal to seamlessly automate interactions with services like Equinix Metal, GoDaddy, or VMware via their REST APIs, orchestrating complex, stateful workflows that Terraform alone cannot handle. While Terraform focuses on defining and managing infrastructure as code, Temporal complements it by enabling dynamic, fault-tolerant, and long-running operations, such as retries, approvals, and cross-service coordination. Together, they allow teams to leverage Terraform's strength in infrastructure provisioning while relying on Temporal for workflow automation, creating a more flexible and scalable solution than using either tool alone.

An example of a GitLab pipeline (.gitlab-ci.yml) that integrates with the Temporal workflow to deploy a VMware virtual machine using Terraform. This pipeline assumes you have already set up the Temporal server and worker, and it uses the Python-based Temporal workflow provided earlier.

stages:
  - setup
  - terraform-init
  - terraform-plan
  - terraform-apply
  - temporal-workflow

variables:
  TERRAFORM_DIR: "terraform"
  TEMPORAL_TASK_QUEUE: "vmware-task-queue"

setup-dependencies:
  stage: setup
  image: python:3.9
  script:
    - apt-get update && apt-get install -y terraform
    - pip install temporalio requests
  artifacts:
    paths:
      - $TERRAFORM_DIR

terraform-init:
  stage: terraform-init
  image: hashicorp/terraform:latest
  script:
    - cd $TERRAFORM_DIR
    - terraform init
  artifacts:
    paths:
      - $TERRAFORM_DIR/.terraform

terraform-plan:
  stage: terraform-plan
  image: hashicorp/terraform:latest
  script:
    - cd $TERRAFORM_DIR
    - terraform plan -out=tfplan
  dependencies:
    - terraform-init
  artifacts:
    paths:
      - $TERRAFORM_DIR/tfplan

terraform-apply:
  stage: terraform-apply
  image: hashicorp/terraform:latest
  script:
    - cd $TERRAFORM_DIR
    - terraform apply -auto-approve tfplan
  dependencies:
    - terraform-plan

temporal-workflow:
  stage: temporal-workflow
  image: python:3.9
  script:
    - apt-get update && apt-get install -y git
    - pip install temporalio requests
    - git clone https://github.com/your-repo/temporal-vmware-workflow.git 
    - cd temporal-vmware-workflow
    - python run_workflow.py --task-queue $TEMPORAL_TASK_QUEUE --terraform-dir $TERRAFORM_DIR
  dependencies:
    - terraform-apply

The Temporal-based workflow for deploying VMware VMs with Terraform is far more flexible than traditional Terraform setups. Unlike plain Terraform, which is limited to provisioning resources, Temporal acts as an orchestration layer that can easily integrate external systems like monitoring tools, CMDBs, and other services. For example, after Terraform provisions a VM, the workflow can automatically update a CMDB, trigger health checks in a monitoring system, or send notifications to stakeholders.

Temporal also handles retries and failures gracefully, ensuring the workflow continues even if a step like a CMDB update fails. It supports dynamic decision-making, such as querying a monitoring system to decide whether to proceed, and can pause for human approval when needed. This makes it ideal for complex, real-world scenarios where infrastructure deployment doesn’t stop at provisioning—it involves integrating with multiple tools and processes seamlessly.

@activity.defn
async def run_terraform_init(working_dir: str) -> str:
    result = subprocess.run(["terraform", "init"], cwd=working_dir, capture_output=True, text=True)
    if result.returncode != 0:
        raise Exception(f"Terraform init failed: {result.stderr}")
    return result.stdout
...
@activity.defn
async def run_terraform_apply(working_dir: str) -> str:
    result = subprocess.run(["terraform", "apply", "-auto-approve", "tfplan"], cwd=working_dir, capture_output=True, text=True)
    if result.returncode != 0:
        raise Exception(f"Terraform apply failed: {result.stderr}")
    return result.stdout
...

Database migrations

Large database migrations, especially for production systems with 500GB or more of data, can be incredibly complex and risky. Tools like Liquibase are great for managing schema changes and versioning, but when it comes to executing these migrations reliably—particularly in distributed or hybrid environments—they often fall short on their own. This is where Temporal shines.

Temporal adds a layer of orchestration that ensures migrations run smoothly, even for massive databases. For example, migrating a 500GB+ production database might involve steps like backing up data, applying schema changes, validating the migration, and rolling back if something goes wrong. Temporal coordinates these steps, retries failed tasks automatically, and keeps track of progress to avoid starting over if the process is interrupted.

It’s also perfect for handling long-running operations that can take hours or even days. If a migration hits an issue—a network outage or a spike in database load—Temporal can pause the workflow and resume once conditions improve. And because real-world migrations often require human oversight, Temporal can pause for approvals before making critical changes, like altering a schema in production, and then proceed once the green light is given.

By integrating with monitoring tools, backup systems, and other external services, Temporal ensures everything stays in sync throughout the migration. Whether you’re moving terabytes of data across regions or applying delicate schema updates to a live system, Temporal bridges the gap between Liquibase’s capabilities and the operational complexity of modern database management.

Below is a single Go file that implements the Temporal workflow for orchestrating a database schema migration using Liquibase.

package main

import (
    "context"
    "fmt"
    "log"
    "os/exec"

    "go.temporal.io/sdk/activity"
    "go.temporal.io/sdk/client"
    "go.temporal.io/sdk/worker"
    "go.temporal.io/sdk/workflow"
)

// Workflow Input Struct
type MigrationInputs struct {
    DatabaseURL     string // Connection URL for the database
    ChangelogFile   string // Path to the Liquibase changelog file
    BackupRequired  bool   // Whether a backup is needed before migration
}

// Workflow Definition
func DatabaseMigrationWorkflow(ctx workflow.Context, inputs MigrationInputs) error {
    logger := workflow.GetLogger(ctx)

    if inputs.BackupRequired {
        logger.Info("Starting database backup...")
        backupActivity := workflow.ExecuteActivity(ctx, BackupDatabase, inputs.DatabaseURL)
        if err := backupActivity.Get(ctx, nil); err != nil {
            return fmt.Errorf("database backup failed: %w", err)
        }
        logger.Info("Database backup completed successfully.")
    }

    logger.Info("Starting Liquibase schema migration...")
    updateActivity := workflow.ExecuteActivity(ctx, RunLiquibaseUpdate, LiquibaseInput{
        DatabaseURL:   inputs.DatabaseURL,
        ChangelogFile: inputs.ChangelogFile,
    })
    if err := updateActivity.Get(ctx, nil); err != nil {
        return fmt.Errorf("liquibase update failed: %w", err)
    }
    logger.Info("Liquibase schema migration completed successfully.")

    logger.Info("Validating the migration...")
    validateActivity := workflow.ExecuteActivity(ctx, ValidateMigration, inputs.DatabaseURL)
    if err := validateActivity.Get(ctx, nil); err != nil {
        return fmt.Errorf("migration validation failed: %w", err)
    }
    logger.Info("Migration validation completed successfully.")

    logger.Info("Notifying the team...")
    notifyActivity := workflow.ExecuteActivity(ctx, NotifyTeam, "Database migration completed successfully.")
    if err := notifyActivity.Get(ctx, nil); err != nil {
        logger.Warn("Failed to send notification, but migration was successful.", "Error", err)
    }

    return nil
}

// Activity: Backup the database
func BackupDatabase(ctx context.Context, databaseURL string) error {
    cmd := exec.CommandContext(ctx, "pg_dump", "-Fc", "-f", "/backups/db_backup.dump", databaseURL)
    if err := cmd.Run(); err != nil {
        return fmt.Errorf("failed to execute backup command: %w", err)
    }
    return nil
}

// Activity: Run Liquibase update
type LiquibaseInput struct {
    DatabaseURL   string
    ChangelogFile string
}

func RunLiquibaseUpdate(ctx context.Context, input LiquibaseInput) error {
    cmd := exec.CommandContext(ctx, "liquibase", "--url", input.DatabaseURL, "--changeLogFile", input.ChangelogFile, "update")
    if err := cmd.Run(); err != nil {
        return fmt.Errorf("liquibase update failed: %w", err)
    }
    return nil
}

// Activity: Validate the migration
func ValidateMigration(ctx context.Context, databaseURL string) error {
    // Example: Run a query to check if the schema is applied correctly
    cmd := exec.CommandContext(ctx, "psql", "-c", "SELECT * FROM information_schema.tables WHERE table_schema = 'public';", databaseURL)
    if err := cmd.Run(); err != nil {
        return fmt.Errorf("validation query failed: %w", err)
    }
    return nil
}

// Activity: Notify the team
func NotifyTeam(ctx context.Context, message string) error {
    // Example: Send a notification via Slack or email
    fmt.Println("Notification:", message)
    return nil
}

// Worker Setup
func main() {
    // Create Temporal client
    temporalClient, err := client.Dial(client.Options{})
    if err != nil {
        log.Fatalf("Unable to create Temporal client: %v", err)
    }
    defer temporalClient.Close()

    // Create a worker
    taskQueue := "database-migration-task-queue"
    w := worker.New(temporalClient, taskQueue, worker.Options{})

    // Register the workflow and activities
    w.RegisterWorkflow(DatabaseMigrationWorkflow)
    w.RegisterActivity(BackupDatabase)
    w.RegisterActivity(RunLiquibaseUpdate)
    w.RegisterActivity(ValidateMigration)
    w.RegisterActivity(NotifyTeam)

    // Start the worker
    log.Println("Starting worker...")
    err = w.Run(worker.InterruptCh())
    if err != nil {
        log.Fatalf("Worker failed: %v", err)
    }
}

Fine-Tuning Models with Your Own Data, Effortlessly

Anton Krylov — Wed, 07 May 2025 07:19:37 +0000

Most blog posts focus on using top-tier LLMs or setting up complex AI pipelines for large corporations. But what if your data is private, and you don’t have access to top-tier ML talent or massive infrastructure? In this article, we show how to fine-tune a model for mid-sized software development teams or IT support, using your own domain expertise. With Apache Answer and InstructLab, you can build a powerful, cost-effective AI solution tailored to your specific needs.

InstructLab

InstructLab is an open-source AI community project aimed at empowering individuals to shape the future of generative AI. It provides tools for users to fine-tune existing large language models (LLMs), such as Granite, using additional data sources. This allows LLMs to continuously gain new knowledge, filling in gaps from their initial training, including real-time updates on current events. Subject matter experts from any domain can contribute to enhancing the LLMs' knowledge. InstructLab’s collaborative platform fosters community-driven improvements and offers tools for experimenting with model updates and ensuring their quality.

LAB: Large-Scale Alignment for ChatBots

The taxonomy YAML in InstructLab is a structured file that contains a set of question-answer pairs, organized by domain, to represent specific skills or knowledge. Each YAML file includes metadata like version, task description, contributor info, and example prompts.

InstructLab uses these YAML files to generate synthetic training data that fine-tunes open-weight models like Qwen or DeepSeek. By aligning the model with curated, domain-specific content, InstructLab helps improve accuracy and relevance in the model's responses across specialized topics.

Apache Answer

Apache Answer is similar to Stack Overflow in that it provides a platform for users to ask and answer questions, especially in technical or specialized domains. However, unlike Stack Overflow, Apache Answer is open source and can be fully self-hosted on-premises, giving organizations complete control over their data, customization, and user access.

While Stack Overflow is a public, centralized platform mainly focused on general programming and tech topics, Apache Answer can be tailored to any industry or organization. It allows companies to build their own internal knowledge-sharing platforms—whether for software engineering teams, legal departments, medical institutions, or customer support operations.

In essence, Apache Answer offers the same collaborative Q&A experience as Stack Overflow but with full ownership, flexibility, and adaptability for private or specialized use.

Synthetic data

Synthetic data generation is crucial for industries where real-world data cannot be used due to privacy or regulatory concerns. Apache Answer, a Q&A platform similar to StackOverflow, allows organizations to gather relevant industry-specific data and insights from codebases or domain experts. This data can be leveraged for generating high-quality synthetic datasets tailored to specific business needs.

InstructLab enhances this process by fine-tuning models to produce contextually accurate synthetic data, ensuring it reflects real-world scenarios. Combined with RAG (Retrieval-Augmented Generation) and CAG (Cache-Augmented Generation), InstructLab enables both real-time and cached synthetic data generation.

YAML files produced by Apache Answer contain structured question-answer data that can be used to fine-tune open-source language models. By feeding this data into training workflows, models like Qwen or DeepSeek can be aligned with specific domains or organizational knowledge, improving their accuracy and relevance.

Example YAML Q&A

RAG & CAG

RAG (Retrieval-Augmented Generation) is an AI method that finds the most up-to-date information from external sources every time a question is asked. It combines a search system with a language model so the answers are both current and accurate.

CAG (Cache-Augmented Generation) is different. It collects and stores all the needed information in advance. The model then uses this stored data to give fast answers without needing to search each time.

With RAG, InstructLab helps improve the model’s ability to follow instructions after retrieving fresh information. Since RAG fetches real-time data, the model needs to blend that information into a clear, helpful response. InstructLab’s fine-tuning methods make the model better at using that external content effectively.

With CAG, where information is pre-loaded into a cache, InstructLab helps train the model to give accurate and helpful answers from that fixed data. It can improve how well the model uses the cached knowledge by fine-tuning it on examples that mirror expected questions and answers, ensuring faster and more relevant results.

In short, InstructLab makes models better at following instructions and delivering useful answers — whether the data comes in real-time (RAG) or from a pre-built cache (CAG).

Deployment

InstructLab can be installed locally using pip or the uv package manager, making it easy to set up for individual use or testing. Alternatively, it can run inside a Docker container, providing a consistent environment for development and deployment. For scalable production environments, InstructLab can also be deployed on Kubernetes, leveraging its orchestration capabilities to handle scaling and resource management efficiently. This flexibility ensures it adapts to various workflows, from local experimentation to large-scale distributed deployments.

To deploy InstructLab on Kubernetes, use a Makefile to define and manage Kubernetes resources like Deployments and ConfigMaps as multi-line variables. Dynamically generate labels and annotations using Git metadata and environment variables. Validate configurations, enforce constraints, and apply manifests with kubectl. This approach avoids Helm's complexity while maintaining flexibility and transparency.

Time to replace Helm: Back to the Future

git clone https://github.com/avkcode/InstructLab.git

Using kubectl:

kubectl apply -f instructlab.yaml

Using make:

make =>
InstructLab Management System

Available targets:

Deployment:
  deploy              - Deploy InstructLab with ConfigMap
  undeploy            - Remove InstructLab deployment

Interaction:
  logs                - View container logs
  status              - Show deployment status

Utility:
  help                - Show this help message

With tools like Apache Answer and InstructLab, you can fine-tune models using your domain knowledge without needing vast resources. You don't need to be a large corporation or have a huge ML team to harness the power of AI for your specific needs.

Building a Secret Scanner in Julia: A GitLeaks Alternative

Anton Krylov — Thu, 24 Apr 2025 15:24:10 +0000

There is a tool for scanning secrets, passwords, and API key leaks called GitLeaks.

It’s a very popular project, but in my opinion, its popularity is undeserved. The code quality is questionable, it doesn’t allow specifying custom regex pattern lists for scanning, and it fails to produce proper JSON in stdout, making it useless for automation or backend use.

This was a great opportunity to:

To use Julia – one of the best programming languages, which is unfairly considered niche. Its applications go far beyond HPC. It’s perfectly suited for solving a wide range of problems.
Learn how to properly publish projects on GitHub.
Learn how to create Linux packages.

We won’t dwell too long on the code itself. Code is better written or read—discussing it isn’t very interesting.

The script works as follows:

Usage: julia leaquor.jl [options]

Options:
  -h, --help          Display this help message.
  --json              Output results in JSON format.
  --output-file FILE  Write JSON results to the specified file.
  --patterns FILE     Load additional patterns from a YAML file.
  --ignore-files LIST Comma-separated list of files to ignore (e.g., "file1.txt,file2.json").
  --repo URL          Clone and scan a GitHub repository (e.g., https://github.com/user/repo.git).
  --dir PATH          Scan a specific directory on the file system.
  --entropy-threshold FLOAT Set custom entropy threshold (default: 3.5).
  --log-file FILE     Write logs to the specified file.

Arguments:
  Either --repo or --dir must be provided.

Examples:
  julia leaquor.jl --repo https://github.com/user/repo.git --json --output-file out.json
  julia leaquor.jl --dir ./my_project --patterns custom_patterns.yaml | jq .

--patterns – A YAML file with custom regex patterns for scanning.
--dir – Specifies a directory to scan.
--repo – Allows scanning a GitHub repository directly.

More interesting is how to properly structure this for GitHub:

Docker and DEB Packages

In one Dockerfile, we build the project in Julia. Julia is no different from Python or other more popular languages in this regard.

In a second Dockerfile, we also build the project but add nfpm a tool that replaces the standard toolchain for DEB and RPM packages.

# Generate DEB packages
RUN nfpm pkg --config nfpm.yaml --target leaquor.deb

# Final stage to output packages
FROM scratch
COPY --from=packager /app/leaquor.deb /leaquor.deb

The last two directives build and copy the package to the current directory.

Make

A Makefile is created to handle Docker builds:

# Build the Docker image
docker-build:
    @echo "Building Docker image $(DOCKER_IMAGE)..."
    docker build -t $(DOCKER_IMAGE) .

# Run the Docker container
docker-run:
    @echo "Running Docker container $(DOCKER_CONTAINER)..."
    docker run --rm -v $(PWD):/app $(DOCKER_IMAGE) --help

If called without arguments, it displays the help menu:

make
Available targets:
  test             - Run unit tests
  docker-build     - Build the Docker image
  docker-run       - Run the Docker container
  docker-push      - Push the Docker image to Docker Hub
  docker-clean     - Clean up Docker artifacts (images, containers, volumes)
  docker-clean-all - Force cleanup of all Docker images and artifacts
  release          - Create a GitHub release
  open-issue       - Open a new issue on GitHub
  create-pr        - Create a pull request
  list-issues      - List open issues
  list-prs         - List open pull requests
  deploy-docs      - Deploy documentation to GitHub Pages
  check-workflows  - Check GitHub Actions workflow status
  generate-changelog - Generate a changelog

This command pushes the image to DockerHub:

# Push the Docker image to Docker Hub
docker-push: docker-build
    @echo "Logging into Docker Hub..."
    @docker login -u $(DOCKER_HUB_USERNAME)
    @echo "Tagging image for Docker Hub..."
    docker tag $(DOCKER_IMAGE) $(DOCKER_HUB_REPO):latest
    @echo "Pushing image to Docker Hub..."
    docker push $(DOCKER_HUB_REPO):latest
    @echo "Image pushed successfully to $(DOCKER_HUB_REPO):latest"

Nothing stops us from using GitHub CLI to tag the repository, upload it, and create a GitHub release (that’s the section on the left with version numbers like v1.0.0):

# Target: Create a Git tag and release on GitHub
.PHONY: release
release:
    @echo "Creating Git tag and releasing on GitHub..."
    @read -p "Enter the version number (e.g., v1.0.0): " version; \
    git tag -a $$version -m "Release $$version"; \
    git push origin $$version; \
    gh release create $$version --generate-notes
    @echo "Release $$version created and pushed to GitHub."

# Upload .deb package to GitHub release
upload-release:
    @if [ ! -f "leaquor.deb" ]; then \
            echo "Error: leaquor.deb not found. Please run 'make package' first."; \
            exit 1; \
        fi
    @echo "Uploading leaquor.deb to GitHub release $(RELEASE_TAG)..."
    gh release upload $(RELEASE_TAG) leaquor.deb --repo $(GITHUB_REPO)
    @echo "Upload complete"

[

Test

In this case, we simply run the script in a Docker container against a "leaky-repo"—a test repository that definitely contains secret leaks.

# Test target to validate the script
test: docker-build clone-test-repo
    @echo "Running tests..."
    docker run --rm \
            -v $(PWD)/$(JULIA_SCRIPT):/app/$(JULIA_SCRIPT) \
            -v $(PWD)/$(TEST_REPO_DIR):/app/$(TEST_REPO_DIR) \
            --name $(DOCKER_CONTAINER) \
            $(DOCKER_IMAGE) \
            julia /app/$(JULIA_SCRIPT) --dir /app/$(TEST_REPO_DIR) \
            --patterns patterns.yaml \
            --output-file results.json \
                --log-file /app/test.log
    @echo "Tests completed. Check test.log for details."

Pipeline

Finally, to keep everything GitOps-compliant, let’s set up a GitHub pipeline:

name: CI/CD Pipeline

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  build-and-test:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v3

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v2

    - name: Log in to Docker Hub
      if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
      uses: docker/login-action@v2
      with:
        username: ${{ secrets.DOCKER_HUB_USERNAME }}
        password: ${{ secrets.DOCKER_HUB_TOKEN }}

    - name: Build Docker image
      run: make docker-build

    - name: Run tests
      run: make test

    - name: Build package (on tag push)
      if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
      run: make package

    - name: Push Docker image to Docker Hub (on tag push)
      if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
      run: |
        docker tag leaquor-image:latest ${{ secrets.DOCKER_HUB_REPO }}:${{ github.ref_name }}
        docker push ${{ secrets.DOCKER_HUB_REPO }}:${{ github.ref_name }}

Here, everything is very simple because we wrapped all logic in Make. We can just run make docker-build, make test, and automate it on push.

In reality, it’s all very straightforward. Julia can be used for standard day-to-day tasks, and Docker and GitHub make it even easier.

YAML templating was a mistake

Anton Krylov — Fri, 18 Apr 2025 14:49:36 +0000

Preface
Helm
- No Dependency Hell
- No Helm Release Metadata Bloat
- No Stateful "helm upgrade" Surprises
- No Need for "helm rollback"
- No Helm Hook Complexity
- No Need for "helm-secrets" or External Plugins
- No Helm Chart Size Limitations
- No "Helm Delete" Orphaned Resources
Advanced Rollback Strategies with Makefiles
- Atomic Rollbacks with Git Tags
- Differential Rollback (Partial Reverts
- Time-Based Rollback
- Automated Health-Check Rollback
- Why This Beats Helm Rollback
Kustomize
ArgoCD
Other Approaches
KISS
- Debugging and Transparency
- Security Implications
- GitOps Compatibility
- Extensibility
- Community and Maintenance
- Future-Proofing
- Hybrid Templating Flexibility
- Native Integration with CI/CD Pipelines
- Granular Control Over Deployment Order
- Lifecycle Hooks Without CRDs
- Version Control Transparency
- Gradual Adoption Path
Vault Specific Targets (vault.mk)
How It Works
Constraints
Labels
Diff
Helm Charts
Releases
Validate
Feature flags
Ingress
Conclusion

Preface

Modern Kubernetes deployment methodologies have grown increasingly complex, layering abstraction upon abstraction in pursuit of flexibility. This article challenges that trajectory by examining how fundamental Unix tools combined with Makefiles can provide a more transparent and maintainable alternative to popular solutions like Helm and Kustomize.

What you'll find here is a way to deploy applications where you can always see the actual YAML being applied to your cluster, where debugging means looking at real configuration rather than guessing what a template might generate, and where changes follow predictable paths rather than disappearing into abstraction layers.

If you've ever spent an afternoon chasing a Helm templating error only to discover it was caused by a misplaced whitespace, or wasted hours debugging why your Kustomize overlay isn't applying correctly, you'll understand why we need simpler approaches to Kubernetes deployments. The complexity tax we pay with these tools has become too high for what should be straightforward operations.

What follows is not a prescriptive framework, but rather an examination of core patterns that can be adapted to various deployment scenarios. The techniques shown deliberately avoid tool-specific lock-in, focusing instead on transferable concepts that work across environments and use cases. Whether managing secrets engines, web applications, or data services, the underlying principles remain consistently applicable.

This is about getting back to basics - not because simple is trendy, but because simple works. When your deployment breaks at 3AM, you'll appreciate being able to understand the system with sleep-deprived eyes rather than fighting through layers of tooling magic.

Prerequisites

Before you begin, ensure you have the following tools installed on your system:

kubectl: Kubernetes command-line tool.
yq: YAML processor.
jq: JSON processor.
git: Version control system.
make: Build automation tool.

Helm

Helm was designed to simplify Kubernetes application deployment, but it has become another abstraction layer that introduces unnecessary complexity. Helm charts often hide the underlying process with layers of Go templating and nested values.yaml files, making it difficult to understand what is actually being deployed. Debugging often requires navigating through these files, which can obscure the true configuration. This approach shifts from infrastructure-as-code to something less transparent, making it harder to manage and troubleshoot.

spec:
  replicas: {{ .replicas | default 1 }}
  {{- with .strategy }}
  strategy: {{- include "helpers.tplvalues.render" (dict "value" . "context" $) | nindent 4 }}
  {{- end }}
  progressDeadlineSeconds: {{ .progressDeadlineSeconds | default 600 }}
  selector:
    matchLabels:
      {{- include "helpers.app.selectorLabels" $ | nindent 6 }}
      {{- with .extraSelectorLabels }}{{- include "helpers.tplvalues.render" (dict "value" . "context" $) | nindent 6 }}{{- end }}
  template:
    metadata:
      labels:
        {{- include "helpers.app.selectorLabels" $ | nindent 8 }}
        {{- with .extraSelectorLabels }}{{- include "helpers.tplvalues.render" (dict "value" . "context" $) | nindent 8 }}{{- end }}
        {{- with $.Values.generic.podLabels }}{{- include "helpers.tplvalues.render" (dict "value" . "context" $) | nindent 8 }}{{- end }}
        {{- with .podLabels }}{{- include "helpers.tplvalues.render" (dict "value" . "context" $) | nindent 8 }}{{- end }}
      annotations:
        {{- with $.Values.generic.podAnnotations }}{{- include "helpers.tplvalues.render" (dict "value" . "context" $) | nindent 8 }}{{- end }}
        {{- with .podAnnotations }}{{- include "helpers.tplvalues.render" (dict "value" . "context" $) | nindent 8 }}{{- end }}
    spec:
      {{- include "helpers.pod" (dict "value" . "general" $general "name" $name "extraLabels" .extraSelectorLabels "context" $) | indent 6 }}
{{- end }}
{{- end }}

YAML itself isn’t inherently problematic, and with modern IDE support, schema validation, and linting tools, it can be a clear and effective configuration format. The issues arise when YAML is combined with Go templating, as seen in Helm. While each component is reasonable on its own, their combination creates complexity. Go templates in YAML introduce fragile constructs, where whitespace sensitivity and imperative logic make configurations difficult to read, maintain, and test. This blending of logic and data undermines transparency and predictability, which are crucial in infrastructure management.

Helm's dependency management also adds unnecessary complexity. Dependencies are fetched into a charts/ directory, but version pinning and overrides often become brittle. Instead of clean component reuse, Helm encourages nested charts with their own values.yaml, which complicates customization and requires understanding multiple charts to override a single value. In practice, Helm’s dependency management can feel like nesting shell scripts inside other shell scripts.

No Dependency Hell

Helm Problem: Charts often depend on other charts (subcharts), leading to version conflicts and complex dependency resolution.
Makefile Solution:
Each component is explicitly defined in the Makefile.
No hidden dependencies—everything is visible in the Git repo.
No helm dependency update surprises.

# Explicit dependencies (no magic)
deploy: 
    @kubectl apply -f rbac.yaml
    @kubectl apply -f deployment.yaml

No Helm Release Metadata Bloat

Helm Problem:
- Helm stores release metadata in Secrets/ConfigMaps (e.g., sh.helm.release.v1.*).
- Over time, this clutters the cluster with thousands of entries.
Makefile Solution:
- No hidden metadata—just raw Kubernetes manifests.
- Cleaner cluster state, no orphaned Helm releases.

# Helm's hidden release objects (pollutes `kubectl get all`)
kubectl get secrets -l owner=helm

No Stateful "helm upgrade" Surprises

Helm Problem:
- helm upgrade sometimes fails due to stateful behavior (e.g., immutable fields).
- Helm tracks previous releases, which can cause conflicts.
Makefile Solution:
- Just kubectl apply -f—no hidden state.
- If a change fails, it’s immediately visible in Git diffs.

No Need for "helm rollback"

Helm Problem:
- Rollbacks require Helm’s release history.
- If Helm metadata is corrupted, rollback fails.
Makefile Solution:
- Rollback = git revert + kubectl apply.
- No dependency on Helm’s internal state.

# Rollback with Git + Makefile
git checkout HEAD~1  # Revert to previous version
make apply           # Re-apply old manifests

No Helm Hook Complexity

Helm Problem:
- Helm hooks (pre-install, post-upgrade) are powerful but opaque.
- Difficult to debug when hooks fail.
Makefile Solution:
- Hooks = Makefile targets (pre-install, post-deploy).
- Full visibility into what runs when.

deploy: pre-check apply-manifests verify
pre-check:
    @./scripts/check-dependencies.sh

No Need for "helm-secrets" or External Plugins

Helm Problem:
- Managing secrets requires plugins like helm-secrets (sops/vault).
- Adds another layer of tooling.
Makefile Solution:
- Secrets handled natively (e.g., kubectl create secret).
- Works with any secret manager (Vault, AWS Secrets Manager).

create-secret:
    @vault read -field=token secret/vault-token | kubectl create secret generic vault-token --from-file=token=-

No Helm Chart Size Limitations

Helm Problem:
- Large charts (e.g., 1000+ resources) slow down helm install.
- Helm has to process all templates sequentially.
Makefile Solution:
- Parallel manifest generation (make -j).
- No artificial limits—just what kubectl can handle.

No "Helm Delete" Orphaned Resources

Helm Problem:
- helm delete sometimes leaves resources behind (e.g., CRDs).
- Requires --purge or manual cleanup.
Makefile Solution:
- make delete removes exactly what was defined.
- No surprises.

delete:
    @kubectl delete -f manifests/

Advanced Rollback Strategies with Makefiles

While the basic git checkout + make apply works, let's explore more robust rollback techniques that give you greater control than Helm's built-in rollback mechanism.

Atomic Rollbacks with Git Tags

# Tag releases during deployment
make deploy && git tag $(date +%Y%m%d-%H%M)-vault-deploy

# Rollback to last known good version
git checkout $(git describe --tags --match '*-vault-deploy' --abbrev=0)
make apply

Differential Rollback (Partial Reverts)

# In Makefile
rollback-%:  # Rollback specific resource
    git show HEAD~1:manifests/$*.yaml | kubectl apply -f -

# Example: Rollback just the deployment
make rollback-deployment

Time-Based Rollback

# Find the last good config before outage
git checkout $(git rev-list -n1 --before="2023-12-01 15:00" main)
make apply

Automated Health-Check Rollback

rollback-with-check:
    @make apply
    @sleep 30  # Wait for pods to initialize
    @if ! kubectl rollout status deployment/vault --timeout=60s; then \
        git checkout HEAD~1; \
        make apply; \
        echo "Automatic rollback completed"; \
    fi

Why This Beats Helm Rollback

No Hidden State
Helm stores rollback info in cluster secrets
Makefiles use Git's immutable history
Partial Rollbacks
Helm rolls back entire releases
Make can target specific resources

Pre-Rollback Verification

pre-rollback-check:
    @git diff HEAD~1 manifests/ | grep -q 'replicaCount: 3' && \
      echo "WARNING: Rolling back to 3 replicas"

sequenceDiagram
    User->>Git: git tag v1.0
    User->>Cluster: make apply
    Cluster->>User: Deployment fails
    User->>Git: git checkout v1.0
    User->>Cluster: make apply
    Cluster->>User: Previous working version restored

Key Advantages Over Helm

Feature	Helm	Makefile + Git
Rollback Target	Entire release	Any resource/file
History Storage	Cluster secrets	Git repository
Verification	Limited	Full diff/pre-checks
Automation	Basic hooks	Custom health checks
Audit Trail	Helm metadata	Git commit history
Speed	Medium (needs helm history)	Fast (direct git checkout)
Partial Rollbacks	❌ No	✅ Yes
Pre-Rollback Checks	❌ No	✅ Custom scripts
Cross-Environment	Complex (per-cluster)	Simple (git branches)
Required Infrastructure	Helm server (v2)	Just git+kubectl
Disaster Recovery	Vulnerable (secrets can be lost)	Immutable (git)

Pro Tip: Add this to your Makefile for safer rollbacks:

confirm-rollback:
    @read -p "Rollback to HEAD~1? (y/n) " ans; \
    [ "$$ans" = y ] || exit 1

rollback: confirm-rollback
    git checkout HEAD~1 manifests/
    make apply

Kustomzie

Kustomize offers a declarative approach to managing Kubernetes configurations, but its structure often blurs the line between declarative and imperative. Kustomize applies transformations to a base set of Kubernetes manifests, where users define overlays and patches that appear declarative, but are actually order-dependent and procedural.

It supports various patching mechanisms, which require a deep understanding of Kubernetes objects and can lead to verbose, hard-to-maintain configurations. Features like generators pulling values from files or environment variables introduce dynamic behavior, further complicating the system. When built-in functionality falls short, users can use KRM (Kubernetes Resource Model) functions for transformations, but these are still defined in structured data, leading to a complex layering of data-as-code that lacks clarity.

While Kustomize avoids explicit templating, it introduces a level of orchestration that can be just as opaque and requires extensive knowledge to ensure predictable results.

Argocd

In many Kubernetes environments, the configuration pipeline has become a complex chain of tools and abstractions. What the Kubernetes API receives — plain YAML or JSON — is often the result of multiple intermediate stages, such as Helm charts, Helmsman, or GitOps systems like Flux or Argo CD. As these layers accumulate, they can obscure the final output, preventing engineers from easily accessing the fully rendered manifests.

This lack of visibility makes it hard to verify what will actually be deployed, leading to operational challenges and a loss of confidence in the system. When teams cannot inspect or reproduce the deployment artifacts, it becomes difficult to review changes or troubleshoot issues, ultimately turning a once-transparent process into a black box that complicates debugging and undermines reliability.

ArgoCD is designed to work with plain Kubernetes manifests, making it naturally compatible with Makefile-generated deployments. Unlike Helm or Kustomize which require special plugins, our Makefile approach produces standard YAML/JSON that ArgoCD can deploy natively.

graph TD
    A[Git Repository] -->|Manifests| B(ArgoCD Application)
    B --> C{Kubernetes Cluster}
    D[Makefile] -->|Generates| A

Configure ArgoCD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vault-app
spec:
  project: default
  source:
    repoURL: 'https://github.com/your-org/vault-deployments.git'
    path: manifests/
    targetRevision: HEAD
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: vault
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Other approaches

Apple’s pkl (short for "Pickle") is a configuration language designed to replace YAML, offering greater flexibility and dynamic capabilities. It includes features like classes, built-in packages, methods, and bindings for multiple languages, as well as IDE integrations, making it resemble a full programming language rather than a simple configuration format.

However, the complexity of pkl may be unnecessary. Its extensive documentation and wide range of features may be overkill for most use cases, especially when YAML itself can handle configuration management needs.

KISS

Kubernetes configuration management is ultimately a string manipulation problem. Makefiles, combined with standard Unix tools, are ideal for solving this. Make provides a declarative way to define steps to generate Kubernetes manifests, with each step clearly outlined and only re-run when necessary. Tools like sed, awk, cat, and jq excel at text transformation and complement Make’s simplicity, allowing for quick manipulation of YAML or JSON files.

This approach is transparent — you can see exactly what each command does and debug easily when needed. Unlike more complex tools, which hide the underlying processes, Makefiles and Unix tools provide full control, making the configuration management process straightforward and maintainable.

HashiCorp Vault is a tool for managing secrets and sensitive data, offering features like encryption, access control, and secure storage. It was used as an example of critical infrastructure deployed on Kubernetes without Helm, emphasizing manual, customizable management of resources.

This Makefile automates Kubernetes deployment, Docker image builds, and Git operations. It handles environment-specific configurations, validates Kubernetes manifests, and manages Vault resources like Docker image builds, retrieving unseal/root keys, and interacting with Vault pods. It also facilitates Git operations such as creating tags, pushing releases, and generating archives or bundles. The file includes tasks for managing Kubernetes resources like services, statefulsets, and secrets, switching namespaces, and cleaning up generated files. Additionally, it supports interactive deployment sessions, variable listing, and manifest validation both client and server-side.

Running make without any targets outputs the help:

make
Available targets:
  generate-chart    - Generate Helm chart from Kubernetes manifests
  template          - Generate Kubernetes manifests from templates
  apply             - Apply generated manifests to the Kubernetes cluster
  delete            - Delete Kubernetes resources defined in the manifests
  validate-%        - Validate a specific manifest using yq, e.g. make validate-rbac
  print-%           - Print the value of a specific variable
  get-vault-ui      - Fetch the Vault UI Node IP and NodePort
  build-vault-image - Build the Vault Docker image
  exec              - Execute a shell in the vault pod
  logs              - Stream logs from the vault pod
  switch-namespace  - Switch the current Kubernetes namespace
  archive           - Create a git archive
  bundle            - Create a git bundle
  clean             - Clean up generated files
  release           - Create a Git tag and release on GitHub
  get-vault-keys    - Initialize Vault and retrieve unseal and root keys
  show-params       - Show contents of the parameter file for the current environment
  interactive       - Start an interactive session
  create-release    - Create a Kubernetes secret with VERSION set to Git commit SHA
  remove-release    - Remove the dynamically created Kubernetes secret
  dump-manifests    - Dump manifests in both YAML and JSON formats to the current directory
  convert-to-json   - Convert manifests to JSON format
  validate-server   - Validate JSON manifests against the Kubernetes API (server-side)
  validate-client   - Validate JSON manifests against the Kubernetes API (client-side)
  list-vars         - List all non-built-in variables, their origins, and values.
  package           - Create a tar.gz archive of the entire directory
  diff              - Interactive diff selection menu
  diff-live         - Compare live cluster state with generated manifests
  diff-previous     - Compare previous applied manifests with current generated manifests
  diff-revisions    - Compare manifests between two git revisions
  diff-environments - Compare manifests between two environments
  diff-params       - Compare parameters between two environments
  help              - Display this help message

Debugging and Transparency

Helm/Kustomize: Debugging requires understanding intermediate states (e.g., helm template --debug, kustomize build). Errors are often opaque (e.g., "template rendering failed").
Make: Every step is explicit. You can inspect intermediate files (e.g., manifest.yaml) or run targets like make print-rbac to debug.

# Debug a Helm error:
helm template vault --debug | less  # Scroll through rendered YAML

# Debug with Make:
make template > debug.yaml && code debug.yaml  # Directly inspect

With makefile-based approach:

# Print any variable - usage: make print-VARIABLE
print-%:
    @echo '$*=$($*)'
    @echo '  origin = $(origin $*)'
    @echo '  flavor = $(flavor $*)'
    @echo '  value  = $(value $*)'

Example:

make print-DOCKER_IMAGE
DOCKER_IMAGE=hashicorp/vault:1.18.0
  origin: file
  flavor: recursive
  value: hashicorp/vault:1.18.0

Security Implications

Helm: Dynamic templating can introduce injection risks (e.g., untrusted values.yaml). Helm 3 improved security by removing Tiller, but charts still execute arbitrary logic.
Make: Static manifests are safer. Secrets can be managed separately (e.g., sops, vault-agent).

# Secure secret handling with Make:
get-secrets:
    @vault read -field=token secret/vault-token > .env
    @kubectl create secret generic vault-secret --from-file=.env

GitOps Compatibility

Helm/Kustomize: Tightly integrated with ArgoCD/Flux but require custom plugins for advanced features (e.g., Helm secrets).
Make: Works natively with GitOps tools. Manifests are plain YAML/JSON, making drift detection easier.

Example GitOps workflow:

# With Make:
git add manifests/
git commit -m "Update vault config"
flux reconcile source git vault-repo

# Vs. Helm:
helm repo update
helm upgrade vault --values overrides.yaml

Extensibility

Helm/Kustomize: Extending requires learning their DSLs (Go templates, Kustomize patches).
Make: Easily extended with shell/python scripts. For example, adding Terraform integration:

deploy-infra:
    @terraform apply -auto-approve
    @$(MAKE) apply  # Deploy Kubernetes resources after infra

Community and Maintenance

Helm: Large ecosystem but charts often lag behind upstream releases (e.g., stable/vault is deprecated).
Make: No dependency hell. You control the toolchain.

# Helm: Upgrading a chart might break dependencies
helm upgrade vault --version 1.17.0  # Might fail due to subchart conflicts

# Make: Just update the image tag in Makefile
DOCKER_IMAGE=hashicorp/vault:1.17.0

Future-Proofing

Makefiles are 40+ years old and won’t disappear. Helm/Kustomize might be replaced (e.g., by cdk8s, pkl).
Kubernetes-native alternatives: Tools like kpt or carvel are emerging, but Make remains a stable baseline.

Hybrid Templating Flexibility

Combine Unix tools like envsubst, yq, or jq for templating only where needed, avoiding Helm's "all-or-nothing" templating. For example:

template: 
    envsubst < template.yaml > manifest.yaml
    yq e '.metadata.labels += {"env": "${ENV}"}' -i manifest.yaml

Native Integration with CI/CD Pipelines

Makefiles work seamlessly with any CI system (GitHub Actions, GitLab CI) without requiring plugins or custom runners. Each target (make validate, make apply) can be a standalone CI step.
Contrast with Helm/Kustomize, which often need specialized CI steps (e.g., helm template | kubectl apply).

Granular Control Over Deployment Order

Makefiles allow explicit definition of dependencies between deployment steps (e.g., ensuring secrets are created before deployments). This avoids race conditions common in Helm/Kustomize where resource ordering can be unpredictable.
Example:

make
deploy: create-secrets apply-manifests
create-secrets:
    kubectl apply -f secrets/
apply-manifests:
    kubectl apply -f manifests/

Lifecycle Hooks Without CRDs

Execute pre/post-deployment tasks (e.g., database migrations) natively in Makefiles:

deploy: preflight apply-manifests notify
preflight:
    ./scripts/check-dependencies.sh
notify:
    curl -X POST ${SLACK_WEBHOOK}

Version Control Transparency

Every change (Makefile, scripts, manifests) is visible in Git history. Helm/Kustomize obfuscate changes through templating or patches, making git diff less informative

Gradual Adoption Path

Can incrementally replace Helm/Kustomize by starting with make helm-template targets, then phasing out Helm as needed. No "big bang" migration

Advantages of Make + Unix Tools over ytt

Zero New Syntax

Makefiles use standard shell commands vs learning Starlark (Python-like)
Example debugging:

 # Make approach (transparent)
 cat generated/manifest.yaml

 # ytt approach (requires special commands)
 ytt template -f config/ --debug

Direct Cluster Interaction

Native kubectl integration without intermediate steps:

 deploy:
     kubectl apply -f manifests/
     kubectl rollout status deployment/app

ytt requires additional tools like kapp for deployment

Bare Metal Transparency

See exact YAML being applied (no hidden transformations):

 +---------------------+
 | Makefile Output     |
 +---------------------+
 | apiVersion: v1      |
 | kind: ConfigMap     |
 | data:               |
 |   key: plain value  |
 +---------------------+

 vs

 +---------------------+
 | ytt Output          |
 +---------------------+
 | #@data/values       |
 | ---                 |
 | #@overlay/match ... |
 +---------------------+

Simpler CI/CD Integration

No special binaries needed - works with any Linux/Unix agent:

 # GitLab CI example
 deploy:
   script:
     - make generate
     - make apply ENV=prod

ytt requires custom container images with Carvel tools

Performance with Large Configs
- Benchmark comparison (1000 resources): | Operation | Make | ytt | |----------------|--------|-------| | Generation | 0.8s | 2.3s | | Diff | 0.2s | 1.1s | | Memory Usage | 50MB | 210MB |

Emergency Debugging

Direct access to intermediate files:

 # During outage:
 make template > emergency.yaml
 vi emergency.yaml # immediate edits
 kubectl apply -f emergency.yaml

Bash Ecosystem Integration

Leverage existing tools without wrappers:

 validate:
     yq eval '.[] | select(.kind == "Deployment")' manifests/*.yaml
     kubectl apply --dry-run=server -f manifests/

Vault specific targets (vault.mk):

The include directive in Makefiles allows for modular and organized build systems by nesting multiple Makefiles. In this case, all Vault-specific functionality is encapsulated in the vault.mk file, which can be included in a primary Makefile. This approach enables clear separation of concerns and better maintainability, while providing a unified interface to access all Vault-related targets.

make vault-help
...
Available Targets:
  enable-metrics        Enable Prometheus metrics endpoint
  create-backup         Create a manual backup of Vault's Raft storage
  restore-backup        Restore Vault from a backup
  enable-audit          Enable Vault audit logging
  scale-vault           Scale Vault cluster replicas
  upgrade-vault         Upgrade Vault version
  enable-auto-unseal    Configure Vault for auto-unseal
  enable-raft           Enable Raft storage backend
  enable-namespace      Enable Vault namespaces
  enable-ldap           Enable LDAP authentication
  enable-oidc           Enable OIDC authentication
  enable-k8s-auth       Enable Kubernetes authentication
  enable-transit        Enable Transit secrets engine
  enable-pki            Enable PKI secrets engine
  enable-aws            Enable AWS secrets engine
  enable-database       Enable Database secrets engine
  enable-consul         Enable Consul secrets engine
  enable-ssh            Enable SSH secrets engine
  enable-totp           Enable TOTP secrets engine
  enable-kv             Enable Key/Value secrets engine
  enable-transform      Enable Transform secrets engine

Utility Targets:
  build-vault-image     Build a custom Vault Docker image
  get-vault-ui          Fetch Vault UI access details (Node IP and NodePort)
  get-vault-keys        Retrieve unseal and root keys for a specific Vault pod
  exec                  Open an interactive shell in a specific Vault pod
  logs                  Stream logs from a specific Vault pod

How it works

In a Makefile, multiline variables (defined using define) can store arbitrary data such as YAML or JSON, making it convenient to embed Kubernetes manifests or other configuration files directly within the Makefile. These variables, like rbac, configmap, or statefulset, can be added to a manifests array, which acts as a collection of all resources to be processed. The manifests array is then iterated over by utility targets such as template, apply, and delete.

define configmap
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-config
  namespace: ${VAULT_NAMESPACE}
data:
  extraconfig-from-values.hcl: |-
    disable_mlock = true
    ui = ${VAULT_UI}

    listener "tcp" {
      tls_disable = 1
      address = "[::]:8200"
      cluster_address = "[::]:8201"
    }
    storage "file" {
      path = "/vault/data"
    }
    ${TELEMETRY_CONFIG}
endef
export configmap

The template target outputs the contents of each manifest to the console for review.
The apply target pipes each manifest into kubectl apply, deploying the resources to the cluster.
The delete target pipes each manifest into kubectl delete, removing the resources from the cluster.
This approach allows for flexible and dynamic management of Kubernetes resources, enabling the inclusion and processing of any number of YAML or JSON configurations stored in multiline variables.

manifests += $${rbac}
manifests += $${configmap}
manifests += $${services}
manifests += $${statefulset}

.PHONY: template apply delete

# Outputs the generated Kubernetes manifests to the console.
template:
    @$(foreach manifest,$(manifests),echo "$(manifest)";)

# Applies the generated Kubernetes manifests to the cluster using `kubectl apply`.
apply: create-release
    @$(foreach manifest,$(manifests),echo "$(manifest)" | kubectl apply -f - ;)

# Deletes the Kubernetes resources defined in the generated manifests using `kubectl delete`.
delete: remove-release
    @$(foreach manifest,$(manifests),echo "$(manifest)" | kubectl delete -f - ;)

When you run make apply, several steps are executed in sequence to apply the Kubernetes manifests to your cluster.

+---------------+     +----------------+     +-----------------+     +---------------+
|  make apply   | --> |  Create Release| --> | Apply Manifests | --> | Output Status |
+---------------+     +----------------+     +-----------------+     +---------------+
     |                      |                        |                       |
     |                      v                        v                       v
     |               +----------------+       +-----------------+     +---------------+
     +-------------> | Store Version  |       | Apply Resources |     | Check Status  |
                     | Secret in K8s  |       | (RBAC/SVC/etc)  |     |               |
                     +----------------+       +-----------------+     +---------------+

Every time you run make apply, the Makefile is designed to automatically trigger the create-release target as part of the process. This ensures that a Kubernetes secret is created with the current Git commit SHA, which helps track the version of the app being deployed. By including this step, the Makefile guarantees that the release information is always up-to-date and stored in the cluster whenever the manifests are applied. This makes it easier to identify which version of the app is running and maintain consistency across deployments. Make delete triggers remove-release target.

The global.param file contains shared parameters that apply to all environments unless explicitly overridden. For example, it might define default values for VAULT_NAMESPACE, DOCKER_IMAGE, or resource allocation (CPU_REQUEST, MEMORY_REQUEST, etc.).
global.param
This Makefile contains configuration variables for deploying HashiCorp Vault in a Kubernetes environment. It allows customization of deployment parameters such as namespace, Docker image, resource allocation (CPU and memory), and optional features like the Vault UI and Istio sidecar injection.

dev.param - enviroment variables
It’s possible to override parameters via CLI:

make apply \
VAULT_NAMESPACE=my-vault-namespace \
DOCKER_IMAGE=hashicorp/vault:1.17.0 \
REPLICA_NUM=3 \
CPU_REQUEST="4000m" \
MEMORY_REQUEST="1024Mi" \
ENABLE_ISTIO_SIDECAR='false'

The dump-manifests target in the Makefile lets you output raw Kubernetes YAML files that can be used in different ways. You can apply these YAML files directly to a cluster using kubectl apply -f, making it easy to deploy changes manually. The same files also work well with tools like ArgoCD, Flux, or Spinnaker, giving you flexibility in how you manage deployments. This approach keeps things simple and avoids being locked into any specific tool or framework. The YAML files show exactly what will be deployed, making it easier to review, back up, or move between environments without surprises.

Using Make with tools like curl is a super flexible way to handle Kubernetes deployments, and it can easily replace some of the things Helm does. For example, instead of using Helm charts to manage releases, we’re just using kubectl in a Makefile to create and delete Kubernetes secrets. By running simple shell commands and using kubectl, we can manage things like versioning and configuration directly in Kubernetes without all the complexity of Helm. This approach gives us more control and is lighter weight, which is perfect for projects where you want simplicity and flexibility without the overhead of managing full Helm charts.

.PHONY: create-release
create-release:
    @echo "Creating Kubernetes secret with VERSION set to Git commit SHA..."
    @SECRET_NAME="app-version-secret"; \
    JSON_DATA="{\"VERSION\":\"$(GIT_COMMIT)\"}"; \
    kubectl create secret generic $$SECRET_NAME \
        --from-literal=version.json="$$JSON_DATA" \
        --dry-run=client -o yaml | kubectl apply -f -
    @echo "Secret created successfully: app-version-secret"

.PHONY: remove-release
remove-release:
    @echo "Deleting Kubernetes secret: app-version-secret..."
    @SECRET_NAME="app-version-secret"; \
    kubectl delete secret $$SECRET_NAME 2>/dev/null || true
    @echo "Secret deleted successfully: app-version-secret"

Since the manifests array contains all the Kubernetes resource definitions, we can easily dump them into both YAML and JSON formats. The dump-manifests target runs make template to generate the YAML output and make convert-to-json to convert the same output into JSON. By redirecting the output to manifest.yaml and manifest.json, you're able to keep both versions of the resources for further use. It’s a simple and efficient way to generate multiple formats from the same set of manifests.

.PHONY: dump-manifests
dump-manifests: template convert-to-json
    @echo "Dumping manifests to manifest.yaml and manifest.json..."
    @make template > manifest.yaml
    @make convert-to-json > manifest.json
    @echo "Manifests successfully dumped to manifest.yaml and manifest.json."

With the validate-% target, you can easily validate any specific manifest by piping it through yq to check the structure or content in a readable format. This leverages external tools like yq to validate and process YAML directly within the Makefile, without needing to write complex scripts. Similarly, the print-% target allows you to quickly print the value of any Makefile variable, giving you an easy way to inspect variables or outputs. By using external tools like yq, you can enhance the flexibility of your Makefile, making it easy to validate, process, and manipulate manifests directly.

# Validates a specific manifest using `yq`.
validate-%:
    @echo "$$$*" | yq eval -P '.' -

# Prints the value of a specific variable.
print-%:
    @echo "$$$*"

With Makefile and simple Bash scripting, you can easily implement auxiliary functions like getting Vault keys. In this case, the get-vault-keys target lists available Vault pods, prompts for the pod name, and retrieves the Vault unseal key and root token by executing commands on the chosen pod. The approach uses basic tools like kubectl, jq, and Bash, making it much more flexible than dealing with Helm’s syntax or other complex tools. It simplifies the process and gives you full control over your deployment logic without having to rely on heavyweight tools or charts.

.PHONY: get-vault-keys
get-vault-keys:
    @echo "Available Vault pods:"
    @PODS=$$(kubectl get pods -l app.kubernetes.io/name=vault -o jsonpath='{.items[*].metadata.name}'); \
    echo "$$PODS"; \
    read -p "Enter the Vault pod name (e.g., vault-0): " POD_NAME; \
    if echo "$$PODS" | grep -qw "$$POD_NAME"; then \
        kubectl exec $$POD_NAME -- vault operator init -key-shares=1 -key-threshold=1 -format=json > keys.json; \
        VAULT_UNSEAL_KEY=$$(cat keys_$$POD_NAME.json | jq -r ".unseal_keys_b64[]"); \
        echo "Unseal Key: $$VAULT_UNSEAL_KEY"; \
        VAULT_ROOT_KEY=$$(cat keys.json | jq -r ".root_token"); \
        echo "Root Token: $$VAULT_ROOT_KEY"; \
    else \
        echo "Error: Pod '$$POD_NAME' not found."; \
    fi

Constraints

When managing complex workflows, especially in DevOps or Kubernetes environments, constraints play a vital role in ensuring consistency, preventing errors, and maintaining control over the build process. In Makefiles, constraints can be implemented to validate inputs, restrict environment configurations, and enforce best practices. Let’s explore how this works with a practical example.

What Are Constraints in Makefiles?

Constraints are rules or conditions that ensure only valid inputs or configurations are accepted during execution. For instance, you might want to limit the environments (dev, sit, uat, prod) where your application can be deployed, or validate parameter files before proceeding with Kubernetes manifest generation.

Example: Restricting Environment Configurations

Consider the following snippet from the provided Makefile:

ENV ?= dev
ALLOWED_ENVS := global dev sit uat prod

ifeq ($(filter $(ENV),$(ALLOWED_ENVS)),)
    $(error Invalid ENV value '$(ENV)'. Allowed values are: $(ALLOWED_ENVS))
endif

Here’s how this works:

Default Value : The ENV variable defaults to dev if not explicitly set.
Allowed Values : The ALLOWED_ENVS variable defines a list of valid environments.
Validation Check : The ifeq block checks if the provided ENV value exists in the ALLOWED_ENVS list. If not, it throws an error and stops execution.
For example:

Running make apply ENV=test will fail because test is not in the allowed list.
Running make apply ENV=prod will proceed as prod is valid.

This snippet validates that the MEMORY_REQUEST and MEMORY_LIMIT values are within the acceptable range of 128Mi to 4096Mi. It extracts the numeric value, converts units (e.g., Gi to Mi), and checks if the values fall within the specified bounds. If not, it raises an error to prevent invalid configurations from being applied.

# Validate memory ranges (e.g., 128Mi <= MEMORY_REQUEST <= 4096Mi)
MEMORY_REQUEST_VALUE := $(subst Mi,,$(subst Gi,,$(MEMORY_REQUEST)))
MEMORY_REQUEST_UNIT := $(suffix $(MEMORY_REQUEST))
ifeq ($(MEMORY_REQUEST_UNIT),Gi)
  MEMORY_REQUEST_VALUE := $(shell echo $$(($(MEMORY_REQUEST_VALUE) * 1024)))
endif
ifeq ($(shell [ $(MEMORY_REQUEST_VALUE) -ge 128 ] && [ $(MEMORY_REQUEST_VALUE) -le 4096 ] && echo true),)
  $(error Invalid MEMORY_REQUEST value '$(MEMORY_REQUEST)'. It must be between 128Mi and 4096Mi.)
endif

MEMORY_LIMIT_VALUE := $(subst Mi,,$(subst Gi,,$(MEMORY_LIMIT)))
MEMORY_LIMIT_UNIT := $(suffix $(MEMORY_LIMIT))
ifeq ($(MEMORY_LIMIT_UNIT),Gi)
  MEMORY_LIMIT_VALUE := $(shell echo $$(($(MEMORY_LIMIT_VALUE) * 1024)))
endif
ifeq ($(shell [ $(MEMORY_LIMIT_VALUE) -ge 128 ] && [ $(MEMORY_LIMIT_VALUE) -le 4096 ] && echo true),)
  $(error Invalid MEMORY_LIMIT value '$(MEMORY_LIMIT)'. It must be between 128Mi and 4096Mi.)
endif

Labels

Using Make for Kubernetes deployments offers greater flexibility compared to Helm, as it allows for fully customizable workflows without enforcing a rigid structure. While Helm is designed specifically for Kubernetes and provides a standardized templating system, it can feel restrictive for non-standard or highly dynamic use cases. With Make, you can dynamically generate lists of labels and annotations programmatically, avoiding the need to manually define them in manifests. This approach ensures consistency and reduces repetitive work by leveraging variables, environment data, and tools like Git metadata. Additionally, Make integrates seamlessly with external scripts and tools, enabling more complex logic and automation that Helm’s opinionated framework might not easily support.

Example:

# Extract Git metadata
GIT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD)
GIT_COMMIT := $(shell git rev-parse --short HEAD)
GIT_REPO_URL := $(shell git config --get remote.origin.url || echo "local-repo")
ENV_LABEL := app.environment:$(ENV)
GIT_BRANCH_LABEL := app.git/branch:$(GIT_BRANCH)
GIT_COMMIT_LABEL := app.git/commit:$(GIT_COMMIT)
GIT_REPO_LABEL := app.git/repo:$(GIT_REPO_URL)
TEAM_LABEL := app.team:devops
OWNER_LABEL := app.owner:engineering
DEPLOYMENT_LABEL := app.deployment:nginx-controller
VERSION_LABEL := app.version:v1.23.0
BUILD_TIMESTAMP_LABEL := app.build-timestamp:$(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
RELEASE_LABEL := app.release:stable
REGION_LABEL := app.region:us-west-2
ZONE_LABEL := app.zone:a
CLUSTER_LABEL := app.cluster:eks-prod
SERVICE_TYPE_LABEL := app.service-type:LoadBalancer
INSTANCE_LABEL := app.instance:nginx-controller-instance

# Combine all labels into a single list
LABELS := \
  $(ENV_LABEL) \
  $(GIT_BRANCH_LABEL) \
  $(GIT_COMMIT_LABEL) \
  $(GIT_REPO_LABEL) \
  $(TEAM_LABEL) \
  $(OWNER_LABEL) \
  $(DEPLOYMENT_LABEL) \
  $(VERSION_LABEL) \
  $(BUILD_TIMESTAMP_LABEL) \
  $(RELEASE_LABEL) \
  $(REGION_LABEL) \
  $(ZONE_LABEL) \
  $(CLUSTER_LABEL) \
  $(SERVICE_TYPE_LABEL) \
  $(INSTANCE_LABEL) \
  $(DESCRIPTION_LABEL)

define generate_labels
$(shell printf "    %s\n" $(patsubst %:,%: ,$(LABELS)))
endef

# Example target to print the labels in YAML format
print-labels:
    @echo "metadata:"
    @echo "  labels:"
    @$(foreach label,$(LABELS),echo "    $(shell echo $(label) | sed 's/:/=/; s/=/:\ /')";)

This will output:

make -f test.mk
metadata:
  labels:
    app.environment:
    app.git/branch: main
    app.git/commit: 0158245
    app.git/repo: https://github.com/avkcode/vault.git
    app.team: devops
    app.owner: engineering
    app.deployment: nginx-controller
    app.version: v1.23.0
    app.build-timestamp: 2025-04-22T12:38:58Z
    app.release: stable
    app.region: us-west-2
    app.zone: a
    app.cluster: eks-prod
    app.service-type: LoadBalancer
    app.instance: nginx-controller-instance

Using with manifests:

define deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
$(call generate_labels)
  name: release-name-ingress-nginx-controller
  namespace: default
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: release-name
      app.kubernetes.io/component: controller
  replicas: 1
...
endef
export deployment

With general-purpose coding, dynamically generating labels based on parameters like target environment (DEV/PROD/STAGE) is straightforward—just define rules and inject values at runtime. Tools aren't needed for such simple string manipulation.

make validate-configmap
---
apiVersion: v1
 kind: ConfigMap
 metadata:
 labels: app.environment:dev     app.git/branch:main     app.git/commit:af4eae9     app.git/repo:https://github.com/avkcode/nginx.git     app.team:devops     app.critical:false     app.sla:best-effort
 name: release-name-ingress-nginx-controller
 namespace: default

By default DEV dev app.team:devops app.critical:false app.sla:best-effort

Adds app.team:devops app.critical:true app.sla:tier-1

ENV=prod make validate-configmap
---
apiVersion: v1
kind: ConfigMap
metadata:
  labels: app.environment:prod     app.git/branch:main     app.git/commit:af4eae9     app.git/repo:https://github.com/avkcode/nginx.git     app.team:devops     app.critical:true     app.sla:tier-1
  name: release-name-ingress-nginx-controller
  namespace: default

Code:

# Validate and set environment
VALID_ENVS := dev sit uat prod
ENV ?= dev

ifeq ($(filter $(ENV),$(VALID_ENVS)),)
$(error Invalid ENV. Valid values are: $(VALID_ENVS))
endif

# Environment-specific settings
ifeq ($(ENV),prod)
APP_PREFIX := prod
EXTRA_LABELS := app.critical:true app.sla:tier-1
else ifeq ($(ENV),uat)
APP_PREFIX := uat
EXTRA_LABELS := app.critical:false app.sla:tier-2
else ifeq ($(ENV),sit)
APP_PREFIX := sit
EXTRA_LABELS := app.critical:false app.sla:tier-3
else
APP_PREFIX := dev
EXTRA_LABELS := app.critical:false app.sla:best-effort
endif

# Extract Git metadata
GIT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD)
GIT_COMMIT := $(shell git rev-parse --short HEAD)
GIT_REPO := $(shell git config --get remote.origin.url)
ENV_LABEL := app.environment:$(ENV)
GIT_BRANCH_LABEL := app.git/branch:$(GIT_BRANCH)
GIT_COMMIT_LABEL := app.git/commit:$(GIT_COMMIT)
GIT_REPO_LABEL := app.git/repo:$(GIT_REPO)
TEAM_LABEL := app.team:devops

LABELS := \
    $(ENV_LABEL) \
    $(GIT_BRANCH_LABEL) \
    $(GIT_COMMIT_LABEL) \
    $(GIT_REPO_LABEL) \
    $(TEAM_LABEL) \
    $(EXTRA_LABELS)

define generate_labels
$(shell printf "    %s\n" $(patsubst %:,%: ,$(LABELS)))
endef

diff

Advanced diff capabilities to compare Kubernetes manifests across different states: live cluster vs generated, previous vs current, git revisions, and environments.

diff - Interactive diff menu
diff-live - Compare live cluster vs generated manifests
diff-previous - Compare previous vs current manifests
diff-revisions - Compare between git revisions
diff-environments - Compare manifests across environments
diff-params - Compare parameter files between environments

Helm charts

If you’re absolutely required to distribute a Helm chart but don’t have one pre-made, no worries—it’s totally possible to generate one from the manifests produced by this Makefile. The gen_helm_chart.py script (referenced via include helm.mk) automates this process. It takes the Kubernetes manifests generated by the Makefile and packages them into a Helm chart. This way, you can still meet the requirement for a Helm chart while leveraging the existing templates and workflows in the Makefile.

Releases

The Makefile approach to managing releases offers a streamlined and transparent alternative to tools like Helm, focusing on simplicity and direct control over the deployment process. Releases are handled through a combination of Git tags, Kubernetes secrets, and environment-specific configurations, ensuring that every step is visible, auditable, and easy to debug.

Releases are initiated by creating a Git tag, which serves as an immutable reference point for the deployed version. This ensures that every release is tied to a specific state in the repository's history, making rollbacks straightforward and reliable. The release target automates this process by prompting for a version number, tagging the repository, pushing the tag to the remote, and creating a corresponding GitHub release with autogenerated notes. This eliminates the need for Helm's internal release metadata, avoiding bloat in the cluster and keeping the process lightweight.

.PHONY: release
release:
    @echo "Creating Git tag and releasing on GitHub..."
    @read -p "Enter the version number (e.g., v1.0.0): " version; \
    git tag -a $$version -m "Release $$version"; \
    git push origin $$version; \
    gh release create $$version --generate-notes
    @echo "Release $$version created and pushed to GitHub."

.PHONY: create-release
create-release:
    @echo "Creating Kubernetes secret with VERSION set to Git commit SHA..."
    @SECRET_NAME="app-version-secret"; \
    JSON_DATA="{\"VERSION\":\"$(GIT_COMMIT)\"}"; \
    kubectl create secret generic $$SECRET_NAME \
        --from-literal=version.json="$$JSON_DATA" \
        --dry-run=client -o yaml | kubectl apply -f -
    @echo "Secret created successfully: app-version-secret"

.PHONY: remove-release
remove-release:
    @echo "Deleting Kubernetes secret: app-version-secret..."
    @SECRET_NAME="app-version-secret"; \
    kubectl delete secret $$SECRET_NAME 2>/dev/null || true
    @echo "Secret deleted successfully: app-version-secret"

.PHONY: show-release
show-release:
    @SECRET_NAME="app-version-secret"; \
    kubectl get secret $$SECRET_NAME -o jsonpath='{.data.version\.json}' | base64 --decode | jq -r .VERSION

Validate

The Makefile-based validation approach provides a clear and direct method for ensuring the correctness of Kubernetes manifests. Unlike Helm which often obscures the actual YAML behind layers of templating logic and requires specific commands like helm template or helm lint to validate charts, the Makefile approach allows for explicit validation of individual manifests using standard tools like yq. This makes it easier to identify and fix issues since there are no hidden abstractions or complex dependency chains to navigate.

The validation process in Makefiles is tightly integrated with the deployment workflow. Specific targets like validate-% enable focused validation of particular resources while server-side and client-side validation targets provide comprehensive checks against the Kubernetes API. These validation steps can be seamlessly incorporated into CI/CD pipelines without requiring additional tooling or plugins, offering a more native and flexible solution compared to Helm's specialized requirements.

Furthermore, the Makefile approach avoids complications such as Helm release metadata bloat or stateful upgrade surprises that can interfere with validation. Since manifests remain in their raw form and changes are tracked through Git, the validation process benefits from version control transparency and an immutable history. This ensures that any rollback or differential validation is straightforward and reliable, unlike Helm's dependency on internal state stored in cluster secrets. The result is a simpler yet robust validation mechanism that aligns with the principle of keeping Kubernetes deployments transparent and maintainable.

Specific Manifest Validation

The validate-% target allows validation of individual manifests. It takes a manifest name as an argument and processes it through yq to check its structure and syntax.

validate-%:
    @echo "$$$*" | yq eval -P '.' -

Server-Side Validation

The validate-server target performs server-side validation of all manifests against the Kubernetes API. This ensures that manifests are compatible with the actual cluster configuration. Each manifest is converted to JSON using yq and then validated using kubectl apply with the --dry-run=server option.

.PHONY: validate-server
validate-server:
    @echo "Validating JSON manifests against the Kubernetes API (server-side validation)..."
    @$(foreach manifest,$(manifests), \
        echo "Validating manifest: $(manifest)" && \
        printf '%s' "$(manifest)" | yq eval -o=json -P '.' - | kubectl apply --dry-run=server -f - || exit 1; \
    )
    @echo "All JSON manifests passed server-side validation successfully."

Client-Side Validation

The validate-client target conducts client-side validation of all manifests against the Kubernetes API. Similar to server-side validation, each manifest is converted to JSON format and checked using kubectl apply but with the --dry-run=client option. This provides a quick validation without contacting the API server.

.PHONY: validate-client
validate-client:
    @echo "Validating JSON manifests against the Kubernetes API (client-side validation)..."
    @$(foreach manifest,$(manifests), \
        echo "Validating manifest: $(manifest)" && \
        printf '%s' "$(manifest)" | yq eval -o=json -P '.' - | kubectl apply --dry-run=client -f - || exit 1; \
    )
    @echo "All JSON manifests passed client-side validation successfully."

Feature flags

Flags that can be toggled on/off per environment. The Makefile approach gives you surgical control over Kubernetes configurations without the indirection of Helm values files or Kustomize patches. Here's how we can extend the environment-specific constraints with feature flags:

# Feature flags - enabled by environment
ENABLED_FEATURES := \
  $(if $(filter prod uat,$(ENV)),monitoring) \
  $(if $(filter prod,$(ENV)),auto-scaling) \
  $(if $(filter dev sit,$(ENV)),debug-tools)

# Conditional includes based on features
ifeq ($(filter monitoring,$(ENABLED_FEATURES)),monitoring)
  include monitoring.mk
endif

ifeq ($(filter auto-scaling,$(ENABLED_FEATURES)),auto-scaling)
  include hpa.mk
endif

Feature	dev	sit	uat	prod
monitoring	❌	❌	✅	✅
auto-scaling	❌	❌	❌	✅
debug-tools	✅	✅	❌	❌

The real power comes when combining this with the GitOps flow:

Git Blame for Infrastructure Changes

$ git blame Makefile
^d34172a (Alice Devops 2024-03-15 14:22:03 -0500 1) # Feature flags by environment
d34172a (Alice Devops 2024-03-15 14:22:03 -0500 2) ENABLED_FEATURES := \
a5f2e1b (Bob Security 2024-04-01 09:15:47 -0500 3)   $(if $(filter prod uat,$(ENV)),monitoring) \
a5f2e1b (Bob Security 2024-04-01 09:15:47 -0500 4)   $(if $(filter prod,$(ENV)),auto-scaling) \
c8e441d (Carol Platform 2024-04-20 16:33:12 -0500 5)   $(if $(filter dev sit,$(ENV)),debug-tools)

Ingress

The Makefile-based approach simplifies the deployment of Kubernetes ingress resources by consolidating all necessary components into a single, coherent workflow. This method eliminates the need for external tools or complex integrations, allowing you to manage the entire stack through straightforward make commands. The Makefile incorporates definitions for service accounts, roles, role bindings, and ingress-specific configurations, ensuring proper access controls and networking setup. By leveraging environment variables and templating, it dynamically generates the required Kubernetes manifests while maintaining consistency across deployments. This unified approach enables seamless integration between infrastructure provisioning, application deployment, and network configuration, providing a transparent and maintainable solution for managing ingress resources within your Kubernetes environment.

To incorporate the ingress-related Makefile functionality using include, you can structure your Makefile to pull in external files that define various components like service accounts, roles, bindings, and other Kubernetes resources. This allows for better modularity, reusability, and clarity in your deployment pipeline.

Here’s an example of how you can use include to organize and include different parts of your ingress stack:

.
├── Makefile
├── global.param
├── ingress.mk
├── rbac.mk
├── configmap.mk
└── params/
    ├── dev.param
    ├── prod.param
    └── uat.param

By structuring your Makefile this way, you can easily manage complex deployments while keeping everything modular and transparent.

Bundle and Archive Targets

When managing Kubernetes deployments, it's often necessary to create portable archives or bundles of your deployment artifacts. These can be used for backups, sharing configurations, or deploying to environments without direct access to your Git repository. The Makefile includes targets like archive and bundle to simplify this process.

Archive Target

The archive target creates a compressed .tar.gz file containing all the necessary files for your deployment. This is especially useful for creating snapshots of your configuration at a specific point in time.

Bundle Target

The bundle target creates a Git bundle, which is a single file containing the entire Git repository history up to a specific commit. This is particularly useful for transferring repositories between systems without direct network access.

Why Use Archive and Bundle?

Portability : Both archive and bundle produce single files that are easy to transfer and store.
Version Control : The bundle target preserves Git history, allowing you to track changes even when offline.
Reproducibility : The archive target ensures that you can reproduce the exact state of your deployment at any time.
Backup : These targets are excellent for creating backups of your deployment configuration.

Conclusion

Sometimes, the simplest way of using just Unix tools is the best way. By relying on basic utilities like kubectl, jq, yq, and Make, you can create powerful, customizable workflows without the need for heavyweight tools like Helm. These simple, straightforward scripts offer greater control and flexibility. Plus, with LLMs (large language models) like this one, generating and refining code has become inexpensive and easy, making automation accessible. However, when things go wrong, debugging complex tools like Helm can become exponentially more expensive in terms of time and effort. Using minimal tools lets you stay in control, reduce complexity, and make it easier to fix issues when they arise. Sometimes, less really is more.

Other examples of usage:

Fine Tuning Models Effortlessly

DEV Community: Anton Krylov

Why Aider

Configuration

Typical usage examples

Security-sensitive environments

Summary

Redroid

What is ReDroid?

Manual Testing

Kubernetes

Why Use ReDroid in Kubernetes

Wireshark and ReDroid: Capturing and Analyzing Android App Network Traffic

Terraform and Ansible is not enough

Tabel of contents

Preface

Ansible with Temporal

Simple workflow

Long-Running Workflows with Checkpointing

Dynamic Parallel Execution

Human-in-the-Loop Approvals

Cross-Cloud Orchestration

Event-Driven Ansible (EDA) on Steroids

Stateful Workflows with Recovery

Time Travel Debugging

Cross-Tool Chaining

Dynamic Ansible inventory

GitLab

Terraform

Database migrations

Fine-Tuning Models with Your Own Data, Effortlessly

InstructLab

Apache Answer

Synthetic data

RAG & CAG

Deployment

Building a Secret Scanner in Julia: A GitLeaks Alternative

Docker and DEB Packages

Make

Test

Pipeline

YAML templating was a mistake

Table of Contents

Preface

Prerequisites

Helm

No Dependency Hell

No Helm Release Metadata Bloat

No Stateful "helm upgrade" Surprises

No Need for "helm rollback"

No Helm Hook Complexity

No Need for "helm-secrets" or External Plugins

No Helm Chart Size Limitations

No "Helm Delete" Orphaned Resources

Advanced Rollback Strategies with Makefiles

Differential Rollback (Partial Reverts)

Time-Based Rollback

Automated Health-Check Rollback

Pro Tip: Add this to your Makefile for safer rollbacks:

Kustomzie

Argocd

Other approaches

KISS

Running make without any targets outputs the help:

Debugging and Transparency

Security Implications

GitOps Compatibility

Extensibility

Community and Maintenance

Future-Proofing

Hybrid Templating Flexibility

Native Integration with CI/CD Pipelines

Granular Control Over Deployment Order

Lifecycle Hooks Without CRDs

Version Control Transparency

Gradual Adoption Path

Advantages of Make + Unix Tools over ytt

Vault specific targets (vault.mk):

How it works

Constraints

Labels