DEV Community: DevOps Start

Secure Terraform PRs with an Architecture Firewall

DevOps Start — Fri, 24 Apr 2026 14:15:37 +0000

Stop the 'merge and pray' workflow! This guide was originally published on devopsstart.com and explores how to implement an automated architecture firewall for your Terraform PRs using OPA.

Introduction

An architecture firewall is a governance layer integrated into your CI/CD pipeline that automatically blocks infrastructure changes violating security or organizational standards before they reach your environment. Unlike a network firewall that filters packets, this firewall filters Pull Requests (PRs). It transforms your infrastructure requirements from passive documentation in a wiki into active, executable code that cannot be ignored.

In this article, you will learn how to move beyond the "merge and pray" workflow by implementing Policy as Code (PaC). We will explore the technical bridge between a terraform plan and automated validation using tools like Open Policy Agent (OPA) and Checkov. You'll discover how to create a pipeline that converts Terraform plans to JSON, evaluates them against strict guardrails and provides immediate feedback to developers via PR comments. By the end, you will have a strategy to enforce encryption, restrict public access and prevent accidental resource deletion without slowing down your engineering velocity. This approach aligns with modern Terraform testing best practices, ensuring that your cloud footprint remains secure by design rather than by chance.

Why Manual PR Reviews Fail the Architecture Test

Relying solely on human peer reviews to catch security holes is a recipe for a production outage. In high-velocity environments, reviewers suffer from fatigue. When a developer submits a PR with 500 lines of HCL, a reviewer might miss a single 0.0.0.0/0 in a security group or a missing encryption_enabled = true flag on an S3 bucket. Humans are great at reviewing logic and intent, but they are terrible at consistently auditing thousands of lines of configuration against a 50-page security compliance PDF.

The "Merge and Pray" workflow creates a dangerous gap where "architectural drift" occurs. This happens when the actual state of your cloud deviates from your intended security posture because a few "small" exceptions were merged over time. To solve this, you need an automated gate that operates on the terraform plan output. This plan is the only source of truth because it represents exactly what Terraform intends to do, accounting for variables, modules and the current state of the cloud.

For example, if you are using Terraform v1.9.0+, you can generate a machine-readable plan that your architecture firewall can analyze. This removes the ambiguity of reviewing the .tf files alone, which doesn't show the final resolved values.

# Generate the binary plan file
terraform plan -out=tfplan

# Convert the binary plan to JSON for policy evaluation
terraform show -json tfplan > tfplan.json

By shifting the audit from the code to the plan, you ensure that the firewall sees the final result, not just the intent. This is the foundation of a robust governance strategy.

Implementing the Policy as Code Engine

To build an architecture firewall, you must choose a Policy as Code (PaC) engine. For simple, industry-standard checks, tools like Checkov or TFLint are excellent because they come with hundreds of pre-built policies. However, for complex organizational logic (such as "Production databases must be deployed in three availability zones and have a specific naming convention"), you need a general-purpose policy engine like Open Policy Agent (OPA). OPA uses a language called Rego to query JSON data.

The technical flow is straightforward: your CI pipeline runs the plan, converts it to JSON and pipes that JSON into OPA. If the Rego policy returns a "deny" result, the CI pipeline fails and the PR is blocked from merging. This turns your security requirements into a unit test for your infrastructure.

Below is a practical example of a Rego policy that prevents any AWS S3 bucket from being created without server-side encryption.

package terraform.analysis

import future.keywords.if

# Default allow unless a violation is found
default allow = true

# Violation: S3 bucket without encryption
deny[msg] if {
    resource := input.resource_changes[_]
    resource.type == "aws_s3_bucket"

    # Check if server_side_encryption_configuration is missing or empty
    # In Terraform JSON, 'after' contains the planned state
    not resource.change.after.server_side_encryption_configuration
    msg := sprintf("Security Violation: S3 bucket %s must have encryption enabled", [resource.address])
}

To run this against your plan in a GitHub Action or GitLab CI runner using OPA v0.60.0, you would execute:

# Run OPA evaluation and capture the deny rules
opa eval -i tfplan.json -d policy.rego "data.terraform.analysis.deny"

If the output contains any messages, the firewall has triggered and the build should fail. This process provides a mathematical guarantee that no unencrypted bucket ever reaches production.

Real-World Application: Preventing Production Catastrophes

A common production nightmare is the accidental deletion of a critical resource, such as a primary database or a core VPC, due to a renaming error or a module refactor. A manual reviewer might not realize that changing a resource name in Terraform results in a "destroy and recreate" action. An architecture firewall can catch this by analyzing the actions array in the terraform plan JSON.

By writing a policy that flags any delete action on resources tagged as critical, you create a safety net. This doesn't mean you can never delete things; it means you must explicitly acknowledge the risk, perhaps through a "break-glass" label on the PR or a manual override from a Lead Architect.

Consider this scenario: a developer changes the name of an RDS instance to match a new naming convention. Terraform sees this as deleting the old DB and creating a new one. Without a firewall, the PR looks like a simple string change. With a firewall, the system sees a delete action on an aws_db_instance and blocks it.

Here is how you would implement a "Protection" rule in Rego to block deletions of production databases:

package terraform.analysis

deny[msg] if {
    resource := input.resource_changes[_]
    resource.type == "aws_db_instance"

    # Check if the action includes 'delete'
    resource.change.actions[_] == "delete"

    # Only apply this to production environments by checking input variables
    input.variables.environment == "prod"

    msg := sprintf("CRITICAL FAILURE: Attempting to delete production database %s. This action is blocked by the Architecture Firewall.", [resource.address])
}

Integrating this into your workflow requires a tight loop. You can use automation for Terraform reviews to post these specific error messages directly as comments on the offending line of the PR. This transforms the "No" from the security team into a helpful, automated suggestion from the platform.

Best Practices for Architecture Guardrails

Implementing a firewall can create friction if handled poorly. If every PR is blocked by 50 different warnings, developers will find ways to bypass the system. Use these strategies to balance security with velocity.

Distinguish Between Warnings and Failures. Not every policy should block a merge. Use "Advisory" levels for things like "missing cost-center tag" (warning) and "Critical" levels for "open SSH port" (hard fail). This prevents the firewall from becoming a nuisance.
Version Your Policies. Treat your Rego or Checkov policies like application code. Store them in a separate Git repository, version them and test them against a suite of "known-bad" Terraform plans to ensure no regressions in your security posture.
Provide Remediation Guidance. A failure message like Policy violation: SEC-01 is useless. Your firewall should return Security Violation: Port 22 is open to 0.0.0.0/0. Please restrict this to the corporate VPN range (10.x.x.x).
Implement an Exception Process. There will always be a legitimate reason to break a rule. Create a standardized way to grant exceptions, such as requiring a specific metadata tag (exception_id = "SEC-123") that the policy engine is programmed to ignore.
Shift Left with Local Pre-commit Hooks. Don't make the CI pipeline the first time a developer sees a failure. Provide a pre-commit configuration using tools like terraform-docs and checkov so they can catch errors on their local machine.

FAQ

Does the architecture firewall replace the need for manual peer reviews?

No, it augments them. The firewall handles the "objective" checks (security, compliance, syntax) so that human reviewers can focus on the "subjective" checks (architecture design, business logic and efficiency). It removes the tedious parts of the review process, allowing engineers to have higher-level discussions about the implementation.

Which tool should I choose: OPA, Checkov, or Sentinel?

If you are using Terraform Cloud/Enterprise, Sentinel is the native choice and offers the deepest integration. If you need a free, industry-standard scanner that works out-of-the-box with minimal configuration, go with Checkov. If you have complex, custom business logic that spans multiple cloud providers and requires a powerful query language, Open Policy Agent (OPA) is the gold standard. I have seen mature platform teams use Checkov for general security and OPA for custom organizational guardrails.

How do I prevent the firewall from slowing down my deployment pipeline?

Running terraform plan and opa eval typically adds less than 60 seconds to a pipeline. To further optimize, you can run these checks in parallel with other tests. Additionally, by implementing local pre-commit hooks, you reduce the number of failed CI runs, meaning the pipeline only handles "clean" code.

Can I use this to manage costs?

Yes, this is a powerful use case. You can write policies that analyze the resource_changes for expensive instance types. For example, you can block any PR that attempts to spin up an aws_instance of type p4d.24xlarge unless the project has a specific "high-compute" approval tag. This prevents "bill shock" by catching expensive mistakes before the resources are actually provisioned.

Conclusion

Building an architecture firewall is about shifting your mindset from "trusting the reviewer" to "trusting the system." By implementing Policy as Code, you ensure that your security standards are consistently applied across every single PR, regardless of who is reviewing it. This creates a scalable governance model that allows your platform team to support hundreds of developers without becoming a bottleneck.

To get started, don't try to automate your entire security handbook at once. Start with the "low hanging fruit": block public S3 buckets and open SSH ports. Once your team is comfortable with the automated feedback loop, gradually introduce more complex architectural rules.

Your next steps are to install OPA or Checkov, integrate a terraform show -json step into your GitHub Actions or GitLab CI and write your first "deny" rule. This transition from manual oversight to automated guardrails is the defining characteristic of a mature Platform Engineering organization.

Local LLM for Log Analysis: Privacy-First Debugging with Ollama

DevOps Start — Fri, 24 Apr 2026 14:05:27 +0000

Stop sending sensitive production logs to the cloud. This guide, originally published on devopsstart.com, shows you how to build a privacy-first debugging stack using Ollama and Llama 3.

Introduction

Sending production logs to cloud AI APIs is a non-starter for any serious SRE in a regulated industry. The answer to maintaining security while gaining AI capabilities is to shift the inference engine to your own hardware. By deploying a local LLM stack using Ollama and Llama 3, you can perform semantic log analysis and root cause diagnosis without a single byte of data leaving your secure perimeter.

Whether you are in fintech, healthcare, or govtech, the "Compliance Wall" is real. You cannot risk leaking PII, session tokens, or internal IP addresses to a third party, even with "Enterprise" privacy agreements. You can find the fundamental concepts of managing these workloads in the official Ollama documentation, which provides the framework for running open-source models locally.

Why this take

Most organizations try to solve the privacy problem with PII Redaction scripts before sending logs to a cloud provider. This is a flawed strategy. Regular expressions and basic NER (Named Entity Recognition) models always miss something. A leaked credit card number or a proprietary internal URL in a stack trace can trigger a compliance audit that costs your company millions. The only way to guarantee zero leakage is to ensure the data never leaves the air-gapped environment or the VPC.

In a production environment with over 500 microservices, the sheer volume of logs makes manual grep-ing impossible. I have seen teams spend six hours correlating logs across three different namespaces just to find a single timeout. A local LLM, when fed a curated slice of logs, can identify the behavioral pattern of a failure in seconds. For example, a sequence of 200 OK responses that occur in an impossible order often indicates a logic bug that regex-based monitors will never catch.

Consider the operational reality of a CrashLoopBackOff. Instead of manually running kubectl logs and kubectl describe and trying to map them in your head, you can pipe the output directly into a local model. When you are Fixing Kubernetes CrashLoopBackOff in Production, the bottleneck is usually the cognitive load of parsing verbose Java or Go stack traces. A local LLM reduces this load by summarizing the failure point immediately.

The cost of cloud tokens for log analysis is astronomical. Logs are verbose. If you send 10MB of logs to a cloud LLM for every incident, your monthly bill will skyrocket. Running a 7B or 8B parameter model on a dedicated GPU node costs nothing but the electricity and the initial hardware investment.

The strongest counter-argument

The most common pushback against local LLMs is the "Hardware Tax." Critics argue that the VRAM requirements for acceptable performance are too high for a standard developer laptop or a typical DevOps jump box. It is true that running a 70B parameter model requires multiple A100s or H100s to be performant, which is an unreasonable ask for a local debugging setup. If you try to run a large model on a CPU with 16GB of RAM, the tokens per second will be so slow that you might as well go back to using grep.

There is also the issue of Context Window limitations. A production log file can be several gigabytes, while most local models have a context window ranging from 8k to 128k tokens. You cannot simply upload a log file to Ollama and ask what happened. You have to implement a pre-processing pipeline to slice the logs, filter out the noise, and feed the model only the relevant window surrounding the timestamp of the error. This adds architectural complexity that a simple API call to OpenAI does not have.

However, these arguments ignore the reality of model quantization. Using 4-bit quantization (GGUF format), you can run a Llama 3 8B model on a machine with as little as 8GB of VRAM with negligible loss in reasoning capability for log analysis. For DevOps tasks, you do not need the creative writing abilities of a 175B parameter model; you need a model that understands stack traces and Kubernetes events.

Exceptions where cloud LLMs still win

There are specific scenarios where a local LLM is the wrong tool. If you are a tiny startup with zero regulatory constraints and no dedicated hardware, the overhead of managing an Ollama instance is a distraction. In those cases, the speed of onboarding a cloud API outweighs the privacy risks.

Cloud LLMs also win when you need cross-domain knowledge at an extreme scale. If your log error is caused by a very obscure bug in a niche third party library that was updated two weeks ago, a cloud model trained on the most recent web crawl might have the answer. A local model's knowledge is frozen at the time of its training.

Additionally, if your team requires a collaborative, multi-user interface with complex permissioning and auditing for every single prompt, building that on top of Open WebUI requires more effort than using a managed SaaS platform. For the senior SRE who needs to diagnose a production outage in a secure environment, these advantages are irrelevant.

Implementing the Privacy-First Stack

To move from theory to production, use Ollama for the backend, Llama 3 (8B) for the reasoning, and Open WebUI for the interface.

Installing the Engine

On a Linux workstation with an NVIDIA GPU, install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Once installed, pull the Llama 3 model. I recommend the 8B version for most log tasks as it balances speed and accuracy.

ollama run llama3:8b

The Log Pipeline Architecture

You cannot dump a 1GB log file into the model. You must use a pipeline. The most effective flow is: Log Source → Grep/Awk Filter → Local LLM.

For example, if you are debugging an OOMKilled pod, first extract the relevant events. If you have already followed the steps to Debug OOMKilled Pods in Kubernetes, you know that the describe output is more valuable than the application logs.

Use this bash script to automate the extraction and analysis:

# Extract the last 100 lines of logs and the pod description
kubectl describe pod my-app-6f7d8-abc > pod_desc.txt
kubectl logs my-app-6f7d8-abc --tail=100 > pod_logs.txt

# Combine them into a prompt file
echo "Act as a Senior SRE. Analyze the following Kubernetes pod description and logs to find the root cause of the failure. Focus on memory limits and exit codes." > prompt.txt
cat pod_desc.txt >> prompt.txt
echo "--- LOGS ---" >> prompt.txt
cat pod_logs.txt >> prompt.txt

# Pipe the prompt to Ollama
cat prompt.txt | ollama run llama3:8b

Prompt Engineering for DevOps

Generic prompts yield generic answers. To get production-ready insights, give the LLM a persona and a specific constraint.

Bad Prompt: "What is wrong with these logs?"

Good Prompt:
"Act as a Site Reliability Engineer specializing in Java Spring Boot applications. I am providing a heap dump summary and the last 50 lines of the application log. Identify if this is a Memory Leak or a sudden spike in traffic. Provide the answer in a bulleted list: 1. Root Cause, 2. Evidence from logs, 3. Recommended fix."

When dealing with complex orchestration issues, such as those found when you Fix CrashLoopBackOff in Kubernetes Pods, use this template:

Persona: Kubernetes Expert
Context: Pod is in CrashLoopBackOff.
Task: Analyze the 'Last State' termination message and the current logs.
Constraint: Ignore health check failures; focus on application-level exceptions.
Logs: [Insert Logs Here]

Hardware Requirements and Performance

The sweet spot for local log analysis is a machine with 24GB of VRAM (like an RTX 3090 or 4090). This allows you to run the 8B model with a massive context window or even experiment with the 70B model using heavy quantization.

Component	Minimum (Fast)	Recommended (Pro)	Note
GPU	NVIDIA RTX 3060 (12GB)	NVIDIA RTX 4090 (24GB)	VRAM is the primary metric
RAM	16GB	64GB	Used for offloading if VRAM fills
Storage	50GB SSD	200GB NVMe	Models are large (4GB to 40GB each)
OS	Ubuntu 22.04	Ubuntu 24.04	Best driver support for CUDA

If you are forced to run on CPU (Apple Silicon M2/M3 is an exception and works great), expect a drop from 50 tokens per second to about 3 to 5 tokens per second. This is acceptable for asynchronous log analysis but frustrating for interactive chatting.

Semantic Anomaly Detection vs. Regex

Standard observability tools like Splunk or ELK rely on indices and keyword searches. If you search for "Error", you find errors. But what if the system is failing silently?

Example: A payment gateway returns 200 OK for every request, but the response body says {"status": "pending", "reason": "timeout"}. A regex monitor sees the 200 and stays green. A local LLM can be prompted to look for logical contradictions:

"Analyze these logs for 'silent failures'. Look for cases where the HTTP status is 200 but the response body indicates a failure or a timeout."

This move from syntactic analysis (looking for patterns) to semantic analysis (understanding meaning) is the real power of the local LLM. It allows you to find the unknown unknowns that you didn't know to write a regex for.

Log Streamlining and Noise Reduction

One of the biggest costs in DevOps is Log Bloat. We store terabytes of INFO logs that we never read. You can use a local LLM as a pre-processor to summarize logs before they are even archived.

By running a small, fast model like Mistral v0.3, you can create a Log Summarizer that takes 1,000 lines of verbose debug logs and converts them into three sentences:

The application started successfully.
It attempted to connect to the database three times and failed.
It entered a sleep state for 30 seconds.

This reduces the cognitive load on the human engineer and can potentially reduce storage costs if you only archive the summaries and a sampled percentage of the raw logs.

Local LLMs are the only viable path for secure, privacy-first debugging in highly regulated environments. While the hardware requirements are higher than using a cloud API, the trade-off is a total elimination of PII leakage risk and the removal of per-token costs. Start by installing Ollama on a GPU-enabled jump box, select a 4-bit quantized Llama 3 model, and begin piping your kubectl outputs into it to reduce your mean time to resolution (MTTR).

How to Build AI Agents for Kubernetes Deployments

DevOps Start — Thu, 23 Apr 2026 14:15:34 +0000

Ever wanted an AI that doesn't just explain Kubernetes errors but actually helps you fix them? This guide, originally published on devopsstart.com, walks through building autonomous K8s agents using MCP, Kagent, and K8sGPT.

Introduction

AI agents for Kubernetes deployments are autonomous systems that follow an "Observe → Reason → Act" loop to resolve cluster issues without manual intervention. While a standard LLM can explain what a CrashLoopBackOff is, a true agent can detect the error, pull the logs, analyze the stack trace, cross-reference it with recent Git commits, and propose a specific PR to fix the environment variable causing the crash.

Building these agents requires moving beyond simple prompting and into "tool use" or "function calling." You are essentially giving an LLM a set of specialized skills (API wrappers) that allow it to interact with your cluster, your GitOps pipeline, and your observability stack. In this guide, you will learn how to architect these skills using the Model Context Protocol (MCP) and frameworks like Kagent and K8sGPT to automate the most tedious parts of Kubernetes operations.

For a deep dive into the foundational concepts of managing the pods these agents will be monitoring, see the guide on Kubernetes for Beginners: Deploy Your First Application.

Prerequisites

Before starting this tutorial, you need a functioning Kubernetes environment and the necessary API access for the LLM. I recommend a development cluster (Kind or Minikube) or a staging namespace in a cloud provider like GKE or EKS to avoid accidental production outages.

You will need the following tools installed on your local machine:

kubectl v1.30+: The standard Kubernetes CLI.
Helm v3.14+: For managing the agent's dependencies.
Python 3.11+: Most agent frameworks, including Kagent and LangChain, require modern Python.
An OpenAI API Key (GPT-4o) or Anthropic API Key (Claude 3.5 Sonnet): Agents require high-reasoning models to avoid hallucinations during tool selection.
K8sGPT v0.12+: For the diagnostic skill set implementation.

You should also have a basic understanding of Kubernetes RBAC. Agents operate as identities within the cluster, and giving them cluster-admin privileges is a security risk. You will need to be comfortable creating ServiceAccounts and RoleBindings to enforce the principle of least privilege.

Overview

In this tutorial, we are building a "Deployment Guardian" agent. This isn't a monolithic script, but a modular system capable of three specific skills:

Automated Diagnostics: Using K8sGPT to scan for misconfigurations and interpreting those errors using an LLM.
Resource Right-Sizing: Analyzing pod resource usage and suggesting updates to the Horizontal Pod Autoscaler (HPA).
GitOps Sync Validation: Monitoring ArgoCD application health and triggering syncs when drifts are detected.

The core of this architecture relies on the Model Context Protocol (MCP). MCP is an open standard that decouples the LLM from the specific implementation of the tool. Instead of writing a custom wrapper for every single kubectl command, MCP allows you to expose a standardized "server" that tells the LLM exactly what tools are available, what arguments they take, and what the expected output format is.

By the end of this guide, you will have an agent that provides the root cause and the exact YAML change needed to fix a deployment, integrated directly into your operational workflow. For those managing the underlying infrastructure of these clusters, understanding how to Deploy an EKS Cluster with Terraform provides the necessary context for where these agents actually reside.

Step 1: Architecting the Agent Loop

Before writing code, you must understand how the agent thinks. A standard LLM request is a linear path: Prompt → Response. An agent loop is circular.

When you ask an agent to "Fix the failing deployment in the staging namespace," it performs the following sequence:

Observation: The agent calls a tool (for example, get_pod_status) to see which pods are failing.
Reasoning: It observes three pods in CrashLoopBackOff and reasons that it needs logs to understand the root cause.
Action: It calls get_pod_logs for one of the failing pods.
Observation: The logs show a java.lang.NullPointerException related to a missing database URL.
Reasoning: It checks the ConfigMap to see if the environment variable is defined.
Action: It calls get_configmap.
Final Response: It concludes the environment variable is missing and suggests the specific kubectl patch command or Git PR.

To implement this, you can use a framework like Kagent, which is built on AutoGen. It treats the "DevOps Engineer" as one agent and the "Kubernetes Cluster" as a tool-providing environment.

Step 2: Implementing the Tooling Layer with MCP

The Model Context Protocol (MCP) is the primary mechanism for production-grade agents. Instead of hardcoding functions into your Python script, you run an MCP server that exposes your Kubernetes API.

First, install the MCP SDK for Python:

pip install mcp

Now, create a simple MCP server that provides a "skill" to get pod events. This is more efficient than giving the LLM raw kubectl access because you can filter the output to only include errors, which reduces token usage and hallucination risk.

# k8s_mcp_server.py
from mcp.server.fastmcp import FastMCP
import subprocess

mcp = FastMCP("K8s-Guardian")

@mcp.tool()
def get_pod_errors(namespace: str) -> str:
    """Fetches only Warning events for pods in a specific namespace."""
    cmd = ["kubectl", "get", "events", "-n", namespace, "--field-selector", "type=Warning"]
    result = subprocess.run(cmd, capture_output=True, text=True)

    if result.returncode != 0:
        return f"Error fetching events: {result.stderr}"

    return result.stdout if result.stdout else "No warning events found."

if __name__ == "__main__":
    mcp.run()

To run this server:

python k8s_mcp_server.py

The LLM now sees get_pod_errors as a capability. When it encounters a deployment failure, it will autonomously decide to call this function rather than guessing. This architectural separation allows you to update the Python "skill" without changing the prompt of the LLM.

Step 3: Configuring Least-Privilege RBAC

Giving an AI agent a kubeconfig with cluster-admin is an unacceptable security risk. If the LLM hallucinates a command like kubectl delete ns --all, the agent will execute it.

You must create a dedicated ServiceAccount with a restricted Role. For our Deployment Guardian, the agent needs to read pods, events, and logs, but it should only be able to "patch" specific resources.

Create a file named agent-rbac.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-ai-agent
  namespace: ai-ops
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: agent-read-write-role
  namespace: staging
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "events", "configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: agent-read-write-binding
  namespace: staging
subjects:
- kind: ServiceAccount
  name: k8s-ai-agent
  namespace: ai-ops
roleRef:
  kind: Role
  name: agent-read-write-role
  apiGroup: rbac.authorization.k8s.io

Apply the configuration:

kubectl create namespace ai-ops
kubectl apply -f agent-rbac.yaml

To connect your agent to this identity, use a token-based approach or a projected volume if the agent runs inside the cluster. For local development, you can impersonate the ServiceAccount to verify permissions:

kubectl get pods -n staging --as=system:serviceaccount:ai-ops:k8s-ai-agent

Step 4: Integrating K8sGPT for Diagnostic Skills

While custom MCP tools are great for specific tasks, K8sGPT provides a powerful set of pre-built diagnostic skills. It scans your cluster for common issues and uses an LLM to explain them.

First, install the K8sGPT CLI:

brew install k8sgpt

Now, authenticate it with your LLM provider:

k8sgpt auth add --backend openai --model gpt-4o

To integrate K8sGPT into your agent's skill set, wrap the k8sgpt analyze command into a tool. This allows the agent to trigger a full cluster scan and reason over the results.

# Adding K8sGPT as a tool in our MCP server
@mcp.tool()
def analyze_cluster_health(namespace: str) -> str:
    """Runs a K8sGPT analysis on the namespace to find errors."""
    cmd = ["k8sgpt", "analyze", "--namespace", namespace]
    result = subprocess.run(cmd, capture_output=True, text=True)
    return result.stdout

When you run this, the output provides a detailed analysis:

$ k8sgpt analyze --namespace staging
[!] Pod 'auth-service-6f7d' is in CrashLoopBackOff
Analysis: The pod is failing because the 'DB_PASSWORD' environment variable is missing.
The application expects this variable to be provided via a Secret.

The agent can now combine this high-level analysis with its own get_configmap tool to find where the secret is missing. This creates a tiered diagnostic approach: K8sGPT finds the "what," and the custom MCP tools find the "how" and "where." If you see these errors frequently, check the Fix Kubernetes CrashLoopBackOff in Production guide for manual remediation steps.

Step 5: Building the Resource Optimization Skill

Resource optimization requires the agent to observe metrics (via Prometheus or Metrics Server) and act on the Horizontal Pod Autoscaler (HPA).

To implement this, your agent needs a tool that can query the Metrics Server:

@mcp.tool()
def get_pod_resource_usage(pod_name: str, namespace: str) -> str:
    """Retrieves CPU and Memory usage for a specific pod."""
    cmd = ["kubectl", "top", "pod", pod_name, "-n", namespace]
    result = subprocess.run(cmd, capture_output=True, text=True)
    return result.stdout

The agent's reasoning logic for optimization follows this pattern:

Trigger: The agent is asked to "Optimize the checkout-service."
Observation: It calls get_pod_resource_usage and sees the pod is consistently using 95% of its memory limit.
Observation: It calls kubectl get hpa and sees the HPA is targeting 50% CPU, but the bottleneck is actually memory.
Reasoning: The agent realizes the HPA should be updated to include memory metrics or the memory limit should be increased.
Action: It proposes a YAML change to the HPA definition.

For a detailed explanation of how HPA works to better tune your agent's prompts, read the Kubernetes HPA Deep Dive.

Step 6: Automating GitOps with ArgoCD Integration

An agent that runs kubectl patch directly creates "configuration drift." The source of truth must always be Git. Therefore, your agent's "Act" phase should target your GitOps tool.

If you are using ArgoCD, give your agent tools to interact with the ArgoCD API or the Git repository. First, ensure you have ArgoCD installed; if not, follow the How to Install Argo CD guide.

Now, create a tool that allows the agent to check the sync status of an application:

@mcp.tool()
def get_argocd_app_status(app_name: str) -> str:
    """Checks if an ArgoCD application is Synced and Healthy."""
    cmd = ["argocd", "app", "get", app_name]
    result = subprocess.run(cmd, capture_output=True, text=True)
    return result.stdout

The "GitOps Loop" for the agent is:

Detect: The agent sees a pod failing in the cluster.
Diagnose: It finds that the image tag v1.2.0 has a bug.
Resolve: It searches for the latest stable image tag in the registry.
Act: Instead of running kubectl set image, it uses a GitHub API tool to create a Pull Request updating the image tag in the Git repository.
Verify: It monitors ArgoCD until the app shows as Synced and Healthy.

This workflow ensures the AI agent remains a part of the governed pipeline. You can learn more about managing these sync policies in the Advanced Argo CD Sync Policies tutorial.

Step 7: Implementing Safety Rails and Human-in-the-Loop (HITL)

To prevent "hallucination-driven outages," you must implement a safety layer between the agent's reasoning and the action.

1. The Dry-Run Constraint

Every tool that modifies the cluster must implement a --dry-run=server flag by default. The agent should first call the tool in dry-run mode and present the proposed change.

@mcp.tool()
def propose_deployment_patch(deployment_name: str, namespace: str, patch_yaml: str) -> str:
    """Proposes a change to a deployment using dry-run."""
    with open("patch.yaml", "w") as f:
        f.write(patch_yaml)

    cmd = ["kubectl", "patch", "deployment", deployment_name, "-n", namespace, "--patch-file", "patch.yaml", "--dry-run=server"]
    result = subprocess.run(cmd, capture_output=True, text=True)
    return f"Proposed Change: {result.stdout}"

2. The Approval Gate (HITL)

The agent must not execute a patch or delete command without manual approval from a human operator, typically via a Slack bot or CLI prompt.

Agent: "I've found that the auth-service is OOMKilled. I propose increasing the memory limit from 256Mi to 512Mi. Should I apply this change? [Yes/No]"
Human: "Yes"
Agent: (Executes the actual patch command)

3. Policy-as-Code (Kyverno/OPA)

A cluster-level policy engine like Kyverno or OPA Gatekeeper should be the final line of defense. For example, a policy that prevents any resource from being deleted in the production namespace, regardless of the requester's identity.

Step 8: Testing and Validating Agent Performance

Treat your agent's skills like production code.

Unit Testing Tools

Test each MCP tool independently. If your get_pod_errors tool fails to parse kubectl output, the LLM will receive garbage and hallucinate a solution.

# Example test for the tool
python -c "from k8s_mcp_server import get_pod_errors; print(get_pod_errors('staging'))"

Scenario-Based Validation (Chaos Engineering)

Test your agent by intentionally breaking things in a sandbox:

Inject a Failure: Delete a Secret that a deployment needs.
Trigger Agent: Ask, "Why is the deployment failing?"
Evaluate:

Did it find the missing secret? (Correctness)
Did it suggest the right fix? (Accuracy)
Did it try to delete the namespace? (Safety)
How many tool calls did it take? (Efficiency)

Token Cost and Latency Tracking

Agents can be expensive. A complex diagnostic loop might call 10 different tools, sending significant context back to the LLM. Use tools like LangSmith or Arize Phoenix to trace the agent's thoughts. If the agent loops infinitely (calling the same tool repeatedly), refine the system prompt to include a "maximum tool call" limit.

Troubleshooting

The Agent "Loops" Infinitely

Symptom: The agent calls get_pod_status repeatedly for 20 turns.
Fix: Update the system prompt: "If a tool returns the same result twice, do not call it again. Instead, try a different diagnostic tool or ask the user for more information."

RBAC "Forbidden" Errors

Symptom: Error from server (Forbidden): pods "my-pod" is forbidden: User "system:serviceaccount:ai-ops:k8s-ai-agent" cannot get resource "pods/log".
Fix: Check your Role definition. pods and pods/log are different resources in Kubernetes. You must explicitly list pods/log in the resources section of your RBAC YAML.

Hallucinated CLI Flags

Symptom: The agent tries to run kubectl get pods --show-all-errors, which is not a real flag.
Fix: Be explicit in your MCP tool description. Instead of "Fetch pods," say "Fetch pods using the exact command kubectl get pods -n {namespace}."

Context Window Overflow

Symptom: The agent "forgets" the initial error after calling several tools.
Fix: Implement "summarization" in your tools. Instead of returning raw kubectl output, filter for the top 5 most relevant errors before sending the text to the LLM.

Conclusion

Building AI agents for Kubernetes is a shift from "writing scripts" to "designing capabilities." By utilizing the Model Context Protocol (MCP), you decouple your agent's reasoning from the underlying API calls, allowing you to iterate on "skills" without breaking the agent's logic.

We have moved from the basic "Observe → Reason → Act" loop to a production-ready architecture featuring least-privilege RBAC, GitOps integration via ArgoCD, and strict human-in-the-loop safety rails.

Actionable Next Steps:

Start Small: Implement one "read-only" skill (like the get_pod_errors tool) and run it in a local Kind cluster.
Secure the Perimeter: Apply the RBAC constraints before moving the agent to a shared development environment.
Implement the Gate: Add a manual approval step for any tool that uses kubectl patch or kubectl delete.
Monitor and Refine: Use a tracing tool to see where your agent is hallucinating and refine your tool descriptions accordingly.

How to Manage Multiple Azure Subscriptions in Terraform

DevOps Start — Mon, 20 Apr 2026 22:01:41 +0000

Managing Hub-and-Spoke architectures in Azure can be a challenge when dealing with multiple subscriptions. This guide, originally published on devopsstart.com, explains how to use Terraform provider aliases to streamline your deployments.

How to Manage Multiple Azure Subscriptions in Terraform

To deploy resources across multiple Azure subscriptions in a single Terraform configuration, you must use provider aliases. By default, the azurerm provider targets only one subscription based on your authentication context. To override this, you define multiple provider blocks, assigning an alias to each and specifying a unique subscription_id.

This pattern is essential for Hub-and-Spoke network architectures. In these environments, central shared services (like Azure Firewall or ExpressRoute) live in a Hub subscription, while application workloads reside in separate Spoke subscriptions. Without aliases, you would be forced to run separate Terraform states and pipelines for every single subscription, which makes cross-subscription networking a manual nightmare.

You can find the complete provider specification in the official Terraform Azure Provider documentation.

Implementing Provider Aliases

To start, you need to configure your providers.tf file. The provider without an alias becomes the default. Any provider with an alias must be explicitly called when defining a resource using the provider meta-argument.

# providers.tf

# Default provider (Spoke Subscription)
provider "azurerm" {
  features {}
  subscription_id = "00000000-0000-0000-0000-000000000000"
}

# Aliased provider (Hub Subscription)
provider "azurerm" {
  alias           = "hub"
  features {}
  subscription_id = "11111111-1111-1111-1111-111111111111"
}

When you create a resource, use the provider argument to tell Terraform which subscription to use. If you omit this, Terraform defaults to the primary provider.

# Deploy a VNet in the Hub subscription
resource "azurerm_virtual_network" "hub_vnet" {
  provider            = azurerm.hub
  name                = "hub-vnet"
  address_space       = ["10.0.0.0/16"]
  location            = "eastus"
  resource_group_name = "hub-rg"
}

# Deploy a VNet in the Spoke subscription (default provider)
resource "azurerm_virtual_network" "spoke_vnet" {
  name                = "spoke-vnet"
  address_space       = ["10.1.0.0/16"]
  location            = "eastus"
  resource_group_name = "spoke-rg"
}

Cross-Subscription Data Referencing

A common production scenario involves fetching an existing resource ID from a Hub subscription to use as a property in a Spoke resource, such as creating a VNet peering. In my experience, this is where most "Resource Not Found" errors occur because the data block defaults to the wrong subscription.

# Fetch Hub VNet ID from the Hub subscription
data "azurerm_virtual_network" "hub_vnet_data" {
  provider            = azurerm.hub
  name                = "hub-vnet"
  resource_group_name = "hub-rg"
}

# Create peering in the Spoke subscription pointing to the Hub
resource "azurerm_virtual_network_peering" "spoke_to_hub" {
  name                      = "spoke-to-hub"
  resource_group_name       = "spoke-rg"
  virtual_network_name      = azurerm_virtual_network.spoke_vnet.name
  remote_virtual_network_id = data.azurerm_virtual_network.hub_vnet_data.id
}

By explicitly assigning provider = azurerm.hub to the data block, Terraform authenticates against the Hub subscription to retrieve the ID before attempting to create the peering in the Spoke subscription.

The Module Provider Gotcha

The biggest mistake engineers make with multi-subscription setups is assuming modules inherit aliases automatically. They do not. If you call a module and it contains azurerm resources, those resources will use the default provider regardless of where the module is called from.

To fix this, you must explicitly pass the aliased provider into the module using the providers map.

module "spoke_workload" {
  source = "./modules/workload"

  # Map the module's internal 'azurerm' provider to the 'hub' alias
  providers = {
    azurerm = azurerm.hub
  }

  vnet_id = data.azurerm_virtual_network.hub_vnet_data.id
}

Inside the module code, do not define a provider block. Just use the standard azurerm resource blocks; the mapping happens at the root level. This ensures your modules remain reusable across different environments. I have seen this fail in clusters with >50 nodes where a missed provider mapping caused a production workload to be deployed into a development subscription, leading to significant security audit failures. To maintain high reliability, consider testing your infrastructure as code.

Best Practices for Naming and Scale

Avoid generic names like azurerm.sub1 or azurerm.secondary. In a production environment with dozens of subscriptions, these names provide zero context and lead to configuration errors. Use functional names that describe the role of the subscription:

azurerm.hub
azurerm.shared_services
azurerm.prod_workload
azurerm.identity_mgmt

In environments with more than 50 subscriptions, managing these aliases in a single providers.tf file becomes brittle. At that scale, I recommend splitting your state files by subscription or using a wrapper tool. This reduces the blast radius of a single terraform apply and decreases the time spent in the "refreshing state" phase, which can otherwise take several minutes.

FAQ

Can I use the same Service Principal for multiple subscriptions?
Yes, as long as that Service Principal has the required RBAC roles (for example, Contributor) across all targeted subscriptions. Terraform handles the switching via the subscription_id field in the provider block.

Do I need to run az account set before running Terraform?
No. When you explicitly define subscription_id in the provider block, Terraform ignores the current active subscription in your Azure CLI session and targets the ID specified in the code.

Does using aliases increase the plan time?
Slightly. Terraform must establish separate API sessions for each provider instance. In very large environments, this can add 10 to 30 seconds to the refresh phase.

Conclusion

Using provider aliases is the only professional way to handle multi-subscription Azure deployments. By separating your Hub and Spoke configurations and explicitly passing providers to your modules, you eliminate the risk of deploying resources to the wrong environment.

Your next steps should be to:

Audit your current providers.tf and rename any generic aliases to functional names.
Check your module calls to ensure providers = { ... } is being used for all non-default subscriptions.
Implement data blocks to automate the linkage between Hub and Spoke resources.

GitHub Actions Security: How to Stop Secret Leaks in CI/CD

DevOps Start — Mon, 20 Apr 2026 21:46:31 +0000

Originally published on devopsstart.com, this guide explores how to eliminate static secrets and harden your GitHub Actions pipelines against credential theft.

Introduction

The fastest way to compromise a production environment isn't by hacking a firewall; it's by stealing a long-lived AWS Access Key leaked in a GitHub Actions log. Secret leakage in CI/CD pipelines is a systemic risk because these pipelines possess the "keys to the kingdom", allowing them to provision infrastructure, modify databases and push code to production.

When secrets leak, they typically happen through three vectors: accidental logging, compromised third-party actions or malicious pull requests from external contributors. To stop this, you must move from static secrets to identity-based authentication using OpenID Connect (OIDC) and implement a strict least-privilege model for your workflow permissions.

In this guide, you will learn how to implement OIDC, the danger of mutable version tags, and how to defend against "pwn-request" attacks. For those managing complex infrastructure, combining these security practices with how to automate terraform reviews with github actions ensures that security is baked into the code review process, not just the execution phase.

The Anatomy of a Secret Leak: Why Your Logs Aren't Safe

GitHub provides a built-in masking feature that replaces known secrets with asterisks (***) in the logs. However, this is a convenience feature, not a security boundary. Attackers can easily bypass masking by encoding the secret. If a developer runs echo $SECRET | base64, the resulting string is no longer the original secret and will not be masked. Any user with read access to the action run can decode it instantly.

Another common leak vector is the "debug dump". When a pipeline fails, developers often add run: env or run: printenv to debug the environment. This prints every single environment variable to the logs. While GitHub tries to mask the secrets, any variable that was dynamically generated or slightly modified during the build process will leak in plain text.

The most dangerous leak comes from the supply chain. If you use a third-party action like uses: some-random-user/setup-tool@v1, you are executing arbitrary code from that user's repository. If that account is compromised, the attacker can update the code in @v1 to curl your environment variables to an external server. Because the action runs with the GITHUB_TOKEN and any secrets you passed to it, the attacker gains full access without leaving a trace in your logs.

Moving from Static Secrets to OIDC

The industry standard for securing cloud access in CI/CD is OpenID Connect (OIDC). Long-lived IAM keys (the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY pair) are liabilities because they never expire and are often stored as static GitHub Secrets. If these leak, they remain valid until you manually rotate them. OIDC replaces these static keys with short-lived, identity-based tokens.

With OIDC, GitHub Actions acts as an Identity Provider (IdP). When a workflow runs, it requests a JWT (JSON Web Token) from GitHub. The workflow then presents this token to the cloud provider (AWS, Azure or GCP). The cloud provider verifies the token's signature and checks if the "claims" (such as the repository name or the branch) match a pre-defined trust relationship. If they match, the provider issues a temporary security token, typically valid for one hour.

To implement this in AWS, you first create an IAM Role with a Trust Policy that trusts the GitHub OIDC provider. Then, use the official aws-actions/configure-aws-credentials action (v4). You must specify permissions: id-token: write in your YAML to allow the runner to request the JWT.

# Example: OIDC Authentication for AWS
name: Secure Deploy
on:
  push:
    branches: [ main ]

permissions:
  id-token: write # Required for requesting the JWT
  contents: read  # Required for checkout

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-oidc-role
          aws-region: us-east-1

      - name: Verify Identity
        run: aws sts get-caller-identity

The output of the last command shows the assumed role, not a static user. If this workflow is compromised, the attacker only has a temporary token that expires quickly, which reduces the blast radius significantly compared to static keys.

Hardening the Supply Chain: The Danger of Mutable Tags

Most DevOps engineers use version tags when referencing actions, such as uses: actions/checkout@v4. This looks clean, but it is a security anti-pattern. Tags in Git are mutable; a maintainer (or an attacker who has hijacked the account) can move the v4 tag to a different, malicious commit. You think you are using a trusted version, but the underlying code has changed without your knowledge.

To eliminate this risk, pin actions to a full-length commit SHA. A SHA is an immutable fingerprint of the code. If the code changes by a single character, the SHA changes. While this makes updating actions more tedious, it is the only way to guarantee that the code you audited is the code running today.

I have seen this fail in clusters with >50 nodes where a single compromised community action allowed an attacker to exfiltrate internal environment variables across dozens of repos. In a production environment with over 100 repositories, manually updating SHAs is a burden. Use a tool like Renovate Bot or Dependabot to automate these updates while keeping them pinned.

# UNSAFE: Using a mutable tag
# If the maintainer changes what @v4 points to, your pipeline is compromised.
- uses: actions/checkout@v4

# SAFE: Using a full-length commit SHA
# This code will NEVER change, regardless of what happens to the repository tags.
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

When pinning, always include a comment noting which version the SHA corresponds to. In clusters where security compliance is strict, such as those running on GKE Autopilot or hardened EKS nodes, this level of granularity is mandatory to pass SOC2 or ISO27001 audits.

Defending Against "Pwn-Requests" and Fork Attacks

One of the most overlooked vulnerabilities in GitHub Actions is the handling of Pull Requests from forks. By default, the pull_request event does not grant secrets to the runner for security reasons. However, developers often find this frustrating when they need to run integration tests that require a database key. To solve this, they use the pull_request_target event.

The pull_request_target event is extremely dangerous. Unlike pull_request, it runs in the context of the base branch (usually main) and has access to secrets. If you have a workflow triggered by pull_request_target that checks out the code from the PR branch and then runs a script, a malicious contributor can modify that script in their fork to echo $SECRET | base64. Since the workflow runs with the base branch's permissions, the attacker steals your production credentials.

To safely handle external contributions, never execute untrusted code from a fork while secrets are present. If you need to run tests on a PR, use the standard pull_request event and utilize "Environment" protections.

# DANGEROUS: Vulnerable to pwn-requests
on:
  pull_request_target:
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4 # This checks out the PR code from the fork
      - run: npm install && npm test # The PR author can change 'npm test' to steal secrets
        env:
          API_KEY: ${{ secrets.API_KEY }}

The correct pattern is to require a manual approval from a maintainer before a workflow can access a protected environment's secrets. This creates a human-in-the-loop firewall that prevents automated credential theft.

Best Practices for CI/CD Hardening

To maintain a secure posture, implement these five practices across every repository in your organization.

Implement a Global Permissions Policy: Start every job with the most restrictive permissions. Use permissions: contents: read by default and only add id-token: write or packages: write when specifically required. This prevents a compromised action from deleting your repository.
Use Environment-Based Secrets: Do not put production secrets in the global "Repository Secrets" section. Create a "Production" environment and assign secrets there. This allows you to enforce "Required Reviewers", meaning no code can access production keys without a senior engineer's sign-off.
Automate Secret Scanning: Integrate Gitleaks or TruffleHog into your pipeline as a pre-commit hook or an initial CI step. These tools look for patterns (like AKIA... for AWS) and fail the build if a secret is detected in the commit history.
Avoid Secret Passing via Env: Instead of passing secrets as environment variables to every step, pass them only to the specific step that needs them. This minimizes the number of processes that have the secret in their memory space.
Rotate Credentials Every 90 Days: Even with OIDC, some legacy systems require static keys. Implement a strict rotation policy. If a key is not rotated regularly, a leak might go undetected for months, giving attackers a permanent backdoor.

FAQ

Does GitHub really mask all my secrets in the logs?

No. GitHub only masks the exact string stored in the secret. If your code transforms the secret (e.g., base64 encoding, URL encoding or splitting the string), the resulting output will not be masked. Never rely on masking as a primary security control.

Why is `pull_request_target` worse than `pull_request`?

pull_request runs in the context of the merge commit and has no access to secrets from the base repository. pull_request_target runs in the context of the base branch and has full access to secrets, meaning any code introduced by a contributor in a fork can access those secrets if the workflow executes that code.

Should I use OIDC for every single cloud provider?

Yes. Every major provider (AWS, Azure, GCP and HashiCorp Vault) now supports OIDC for GitHub Actions. Moving away from static JSON keys or CSV credential files reduces your operational overhead and eliminates the risk of "stale" credentials living in your repository settings.

Can I still use version tags like `@v4` if I use a private runner?

Yes, but it is still a bad practice. Even on a private runner, a compromised third-party action can exfiltrate data from your internal network or steal the GITHUB_TOKEN to modify your source code. The location of the runner does not protect you from supply chain attacks.

Conclusion

Securing GitHub Actions requires moving away from the "trust by default" mindset. The combination of OIDC for identity, SHA pinning for supply chain integrity and strict permissions blocks creates a defense-in-depth strategy. The most critical immediate step you can take is auditing your workflows for pull_request_target and replacing static cloud keys with OIDC roles.

Start by implementing these three actionable steps today: first, replace all v* tags with commit SHAs in your most critical deployment pipeline. Second, migrate your production cloud authentication to OIDC to eliminate long-lived keys. Third, configure GitHub Environments with mandatory reviewers for all production secrets. By shifting security left into your CI/CD configuration, you ensure that your pipeline is a tool for delivery, not a liability.

Cursor vs Copilot vs Cody: Best AI Editor for DevOps

DevOps Start — Mon, 20 Apr 2026 21:41:26 +0000

Choosing the right AI editor for DevOps is about more than just autocomplete—it's about codebase context. Originally published on devopsstart.com, this guide compares Cursor, Copilot, and Cody for IaC and Kubernetes workflows.

Introduction

Choosing an AI code assistant for DevOps isn't about who can write the cleanest Python function; it's about who understands the relationship between your variables.tf, your Helm charts and your GitHub Actions workflow. Most AI tools are built for application developers, which means they often fail when faced with the fragmented nature of infrastructure. If you've ever had Copilot suggest a deprecated Terraform provider or a Kubernetes API version that hasn't existed since 1.16, you know the "context problem" firsthand.

In this guide, you'll learn how to navigate the trade-offs between GitHub Copilot, Cursor and Sourcegraph Cody specifically through the lens of a Platform or DevOps engineer. We will dive into how each tool handles codebase indexing, how they manage the hallucinations common in YAML and HCL, and which one actually helps you reduce "time to first green build" in a complex CI/CD pipeline. By the end, you'll have a clear decision matrix to determine which tool fits your specific organizational scale, security requirements and infrastructure complexity. Whether you are managing a handful of scripts or a massive polyglot monorepo, the right choice depends on how the AI "sees" your architecture.

The Context Problem: Why General AI Fails DevOps

DevOps engineers don't write linear code; they build distributed systems. A single change in a Terraform module might require updates to a Kubernetes manifest and a corresponding change in a CI pipeline. Standard AI completions fail here because they typically rely on "active tab" context. If you are editing deployment.yaml but the relevant environment variable is defined in terraform/outputs.tf (which is closed), the AI is guessing based on generic internet patterns, not your actual architecture.

For example, imagine you are trying to reference a secret created by an ExternalSecrets operator. A generic AI will suggest a standard Kubernetes Secret syntax. A context-aware AI knows you are using ExternalSecret objects and will suggest the correct API group. This is the difference between a tool that saves you five seconds of typing and a tool that prevents a production outage. To solve this, tools have moved toward Retrieval-Augmented Generation (RAG), which indexes your local or remote files to provide actual project awareness. You can read more about the complexities of managing these environments in the Kubernetes v1.36 Features, Deprecations & Upgrade Guide to see why version-specific context is so critical.

Consider this scenario: you need to add a new resource to a Terraform module that already has a strict naming convention and specific tagging requirements defined in a separate locals.tf file.

# locals.tf
locals {
  common_tags = {
    Environment = var.env
    Project     = "Phoenix"
    ManagedBy   = "Terraform"
  }
}

# main.tf
# You start typing: resource "aws_s3_bucket" "logs" {
# A context-blind AI suggests: tags = { Name = "logs" }
# A context-aware AI suggests: tags = local.common_tags

When the AI knows your locals.tf exists, it stops hallucinating generic tags and starts following your internal standards. This eliminates the manual "copy-paste" cycle that often leads to inconsistent infrastructure and failed compliance checks.

Cursor: The AI-Native Powerhouse for IaC

Cursor is not a plugin; it is a fork of VS Code. This architectural choice is a game changer for DevOps engineers because it allows the AI to integrate deeply with the IDE's indexing engine. While Copilot feels like a sophisticated autocomplete, Cursor feels like a pair programmer that has actually read your entire repository. It uses a local index of your files, meaning when you ask it to "Add a new environment to the staging cluster," it scans your existing .tfvars and kustomize overlays to mirror the pattern exactly.

For those managing complex Terraform projects, Cursor's @Codebase feature is indispensable. You can prompt the AI to analyze the relationship between different modules without opening every file. This is particularly useful when you are implementing Terraform Testing Best Practices and need the AI to generate test cases based on the actual resource dependencies. In clusters with >100 nodes, where naming conventions are strict and dependencies are deep, this level of indexing prevents the "hallucinated resource" error that plagues plugin-based assistants.

Here is how you would actually use Cursor to refactor a Kubernetes manifest to use a new ConfigMap source:

# In Cursor, you use the Cmd+K (or Ctrl+K) interface.
# Prompt: "@Codebase update all deployments in /k8s/overlays/prod to use the 
# new configmap-v2 defined in configmap.yaml"

# Cursor identifies all files in the directory and applies the change:
# Before:
# configMapRef:
#   name: app-config
# After:
# configMapRef:
#   name: app-config-v2

The magic here is that Cursor doesn't just find and replace text; it understands that the configMapRef is a Kubernetes object property. It maintains the indentation of your YAML (which is the bane of every DevOps engineer's existence) and ensures that the change is consistent across all target files. This removes the tedious manual verification usually required after a bulk edit.

Sourcegraph Cody: Mastering the Enterprise Monorepo

While Cursor excels at local indexing, Sourcegraph Cody is designed for the enterprise scale. Many Platform teams work in massive polyglot monorepos where the Terraform code is in one directory, the Go-based operator is in another and the documentation is in a separate Wiki or GitHub Pages site. Cody's strength lies in its ability to pull context from remote repositories and external documentation via the Sourcegraph index.

Cody is the "Enterprise Context King" because it doesn't just look at your open files; it looks at your entire organization's knowledge graph. If your company has a proprietary way of handling VPC peering or a specific wrapper around Pulumi, Cody can be configured to prioritize those internal patterns over generic public documentation. This is vital for SOC2 or HIPAA compliant environments where "following the internal standard" is not a suggestion, but a legal requirement.

Imagine you are tasked with updating a CI pipeline using a custom internal GitHub Action that isn't documented on the public web.

# .github/workflows/deploy.yml
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Internal Deploy
        uses: my-corp/deploy-helper@v2 # Cody knows this action exists in your org
        with:
          cluster_id: ${{ secrets.CLUSTER_ID }}
          # Cody suggests the 'environment' input because it indexed 
          # the 'deploy-helper' repo in the same organization.
          environment: 'production'

By indexing the my-corp/deploy-helper repository, Cody provides suggestions for inputs and outputs that GitHub Copilot would simply guess. This reduces the need to constantly switch between your editor and the internal documentation browser. For teams implementing GitOps Testing Strategies, Cody can help bridge the gap between the ArgoCD configuration and the underlying Kubernetes manifests by tracing the logic across different repositories.

Comparing AI Performance on YAML and HCL

When it comes to Infrastructure as Code (IaC), the biggest risk is the "confidently wrong" suggestion. HCL (HashiCorp Configuration Language) and YAML are whitespace-sensitive and schema-dependent. GitHub Copilot is generally the fastest for simple snippets, but it is the most prone to hallucinating API versions. For example, it might suggest apiVersion: extensions/v1beta1 for an Ingress resource, which has been deprecated for years.

Cursor and Cody perform better here because they can be anchored to specific versions of your codebase. If your project specifies Terraform v1.7.0 in a .terraform-version file, Cursor is more likely to suggest syntax compatible with that version. In a head-to-head comparison for generating a complex Kubernetes NetworkPolicy, Cursor typically wins on formatting, while Cody wins on referencing your existing network architecture.

Let's look at a practical comparison of how these tools handle a request to create a Kubernetes Service of type LoadBalancer with specific cloud annotations for AWS.

# Prompt: "Create a LoadBalancer service for the 'api' deployment with AWS NLB annotations"

# Copilot: Often gives a generic LoadBalancer without the specific 
# service.beta.kubernetes.io/aws-load-balancer-type: nlb annotation.

# Cursor: Checks your other services, sees you use 'nlb-ip' mode, and suggests:
# annotations:
#   service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"
#   service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

# Cody: References the official AWS Load Balancer Controller docs (if indexed)
# and suggests the most current annotation for your specific K8s version.

The "hallucination risk" in Kubernetes is particularly high because the API evolves so rapidly. A tool that relies on a training set from 2022 will lead you toward deprecated fields. A tool that uses RAG to look at your current kubectl version or your manifest files will guide you toward the current standard.

Best Practices for AI-Driven DevOps

To get the most out of these tools without introducing security vulnerabilities or infrastructure drift, you must treat AI output as a "proposed change" rather than "final code." Follow these guidelines to maintain stability.

Use Version Pinning in Prompts: Never just ask for a "Terraform script." Specify the version. Use prompts like "Using Terraform v1.7.x and the AWS provider v5.0, create a VPC..." This forces the AI to narrow its search space and reduces the likelihood of deprecated syntax.
Verify with Static Analysis: AI is great at writing code but terrible at verifying it. Always pipe AI-generated HCL through terraform validate and YAML through kube-linter or datree. This catches the small indentation errors that AI frequently introduces.
Context-Seed Your Prompts: In Cursor or Cody, explicitly tag the files that define your architecture. Instead of "Fix this error," use "@variables.tf @main.tf fix the mismatch in the subnet ID." This provides the RAG engine with a direct path to the answer.
Sanitize Secrets Before Indexing: Ensure your .gitignore is robust. While most modern AI editors respect .gitignore, double-check that you aren't indexing .terraform.lock.hcl or temporary state files that might contain sensitive metadata.
Iterative Refinement: Start with a high-level architecture prompt, then drill down into specific resources. Asking an AI to "Write my entire EKS cluster" usually results in a mess. Ask it to "Define the VPC," then "Define the EKS cluster using that VPC," and finally "Add the node groups."

FAQ

Which AI editor is the most secure for corporate code?

Sourcegraph Cody generally leads in enterprise security because it offers robust controls over where data is stored and how it is indexed. For organizations with strict data residency requirements, Cody's ability to run on-premises or in a private cloud is a major advantage. Cursor and Copilot have "Privacy Modes" that promise not to train on your data, but for SOC2/HIPAA environments, the transparency of Cody's indexing layer is typically more acceptable to security auditors.

Can these tools actually replace writing Terraform by hand?

No, and attempting to do so is dangerous. AI is excellent at boilerplate (creating 10 similar S3 buckets) and translation (converting a Helm chart to a Kustomize overlay), but it cannot reason about your business logic or the cost implications of a specific instance type. Use AI to handle the "syntax toil" while you handle the "architectural intent."

How do I stop the AI from suggesting deprecated Kubernetes APIs?

The best way is to provide a "source of truth" file in your repository. Create a K8S_STANDARDS.md file that lists your cluster version and preferred API versions. In Cursor or Cody, refer to this file using @K8S_STANDARDS.md in your prompt. This overrides the AI's general training data with your specific project requirements.

Does using a fork like Cursor break my VS Code extensions?

Since Cursor is a fork of VS Code, it is compatible with almost all VS Code extensions. You can import your existing themes, keybindings and plugins (like the HashiCorp Terraform extension) directly. The primary difference is the built-in AI layer, which replaces the need for a separate Copilot plugin.

Conclusion

The transition from "AI as a plugin" to "AI as an environment" is the most significant shift in DevOps productivity since the rise of GitOps. GitHub Copilot remains a solid choice for generalists who want a low-friction experience. However, for the specialized needs of a Platform Engineer, Cursor's local codebase indexing provides a level of precision in HCL and YAML that plugins cannot match. For those operating at a massive corporate scale, Sourcegraph Cody's remote context capabilities make it the only viable choice for navigating polyglot monorepos.

Your next step should be a two-week trial: install Cursor for your local feature development to see if the @Codebase indexing reduces your context-switching. Simultaneously, if you are in a large team, evaluate Cody's ability to index your internal documentation. Once you've chosen your tool, integrate a static analysis step into your CI pipeline to ensure that AI-generated speed doesn't come at the cost of production stability. Stop fighting with YAML indentation and start leveraging the context of your entire architecture.

Build an Internal Developer Platform with Backstage and

DevOps Start — Mon, 20 Apr 2026 21:36:22 +0000

Stop the 'ticket-ops' madness! This guide, originally published on devopsstart.com, shows you how to combine Backstage and Crossplane to build a true self-service Internal Developer Platform.

Introduction

Stop forcing your developers to learn the intricacies of cloud provider consoles or struggle with 500-line Terraform modules just to get a database. The gap between raw infrastructure and developer productivity is where "ticket ops" thrives, slowing down deployment cycles and frustrating engineers. To solve this, you need an Internal Developer Platform (IDP) that abstracts infrastructure complexity into a self-service experience.

An IDP allows developers to provision resources via a simplified interface without needing to be cloud experts. In this guide, you will learn how to build a production-ready IDP by combining Backstage and Crossplane. Backstage acts as your front-end portal, providing a unified interface for service discovery and software templates. Crossplane serves as the back-end control plane, turning Kubernetes into a universal API for managing cloud resources.

By the end of this article, you will understand the architecture required to move from manual Infrastructure as Code (IaC) to a scalable Infrastructure as a Service (IaaS) model. You'll see exactly how to map a button click in a UI to a live AWS RDS instance via GitOps, reducing the cognitive load on your developers while maintaining strict governance for your platform team. For more on managing the underlying clusters, you can check out Kubernetes for Beginners: Deploy Your First Application.

The Architecture: Connecting Backstage to Crossplane

Building an IDP isn't about one tool; it's about the pipeline. The most common mistake is trying to connect Backstage directly to a cloud API. That is a security nightmare and lacks auditability. Instead, use a GitOps-driven control plane architecture. In this flow, Backstage doesn't "create" the infrastructure; it "requests" it by committing a manifest to Git.

The sequence works as follows: a developer selects a "Provision Postgres" template in the Backstage Scaffolder. Backstage then triggers a commit of a simple YAML file to a Git repository. An automated GitOps controller, such as ArgoCD, detects this change and syncs the manifest to a Kubernetes cluster. Inside that cluster, Crossplane v1.14.x sees the new Custom Resource (CR) and communicates with the cloud provider's API to provision the actual resource.

This ensures that your Git history is the single source of truth, which is critical for compliance and disaster recovery. To ensure these deployments are handled reliably, you should learn How to Set Up Argo CD GitOps for Kubernetes Automation.

The "connective tissue" here is the YAML schema. Backstage must output a manifest that exactly matches the CompositeResourceDefinition (XRD) you've defined in Crossplane. If the Scaffolder outputs db_size: small but Crossplane expects storageClass: small, the request will hang in a "Pending" state. You must treat your XRDs as the API contract between your platform team and your developers.

Abstracting Cloud Complexity with Crossplane Compositions

If you give developers raw Crossplane resources, you've just traded Terraform for Kubernetes YAML, which does not reduce cognitive load. The real power of Crossplane lies in Compositions. A Composition allows you to bundle multiple low-level resources (like a VPC, a Subnet, and an RDS instance) into a single, high-level "Composite Resource" (XR) that developers can actually understand.

For example, instead of requiring a developer to specify db.aws.upbound.io/v1beta1 with 20 mandatory fields, you create a CompositeDatabase definition. The developer only provides a name and a size. Your platform team defines the "blueprint" that maps size: small to a t3.micro instance with 20GB of encrypted GP3 storage.

Here is an example of a simplified CompositeResourceDefinition (XRD) that defines the API your developers will use:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xpostgresdatabases.platform.example.org
spec:
  group: platform.example.org
  names:
    kind: XPostgresDatabase
    plural: xpostgresdatabases
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              storageGb:
                type: integer
              region:
                type: string

And here is how the developer's request (the "Claim") looks. This is the exact YAML that Backstage will generate:

apiVersion: platform.example.org/v1alpha1
kind: PostgresDatabase
metadata:
  name: order-service-db
  namespace: order-service-prod
spec:
  storageGb: 20
  region: us-east-1

By using this approach, you eliminate the need for developers to know AWS-specific jargon. You can change the underlying instance type or backup policy in the Composition without ever touching the developer's manifest.

Implementing the Backstage Scaffolder for Self-Service

The Backstage Scaffolder is the engine that turns a user's form input into a Git commit. To make this work with Crossplane, you create a template.yaml file. This template defines the UI form (the questions you ask the developer) and the "steps" required to process the answer.

In a production setup, your template should not just create a file; it should validate the input. For example, if a developer requests 10,000GB of storage, your template or a validating admission webhook in Kubernetes should catch it. The template uses "Nunjucks" templating to inject the form values into the Crossplane Claim YAML.

Below is a snippet of a Backstage software template designed to provision a Crossplane database:

apiVersion: backstage.io/template/scaffolder-entity/v1.0.0
metadata:
  name: provision-rds-postgres
  title: Provision RDS Postgres
  description: Creates a production-ready Postgres DB via Crossplane
spec:
  parameters:
    - title: Database Details
      properties:
        dbName:
          type: string
          title: Database Name
        storageGb:
          type: integer
          title: Storage Size (GB)
          default: 20
        environment:
          type: string
          title: Environment
          enum: [dev, staging, prod]
  steps:
    - id: fetch-base
      action: fetch:template
      input:
        templateRepo: templates/infrastructure/rds
        values:
          name: ${{ parameters.dbName }}
          storage: ${{ parameters.storageGb }}
          env: ${{ parameters.environment }}
    - id: publish
      action: publish:github
      input:
        allowedStatuses: [success]
        repoUrl: github.com?owner=my-org&repo=${{ parameters.dbName }}-infra

When the developer clicks "Create," Backstage creates a new repository (or updates an existing one) with the resulting YAML. The critical part is the fetch:template step. It takes the generic claim.yaml from your template repository and fills it with the user's specific requirements. This removes the possibility of syntax errors in the YAML, as the developer never actually writes the code.

The GitOps Feedback Loop and Production Gotchas

A major pain point in IDPs is the "black hole" effect: a developer clicks a button in Backstage, the commit happens, and then nothing. They have no idea if the database is actually ready or if the Crossplane provider is stuck in a back-off loop. To solve this, you must implement a feedback loop.

One effective method is using the Backstage Kubernetes plugin combined with the Crossplane status fields. Crossplane updates the status section of the Claim resource once the cloud provider confirms the resource is Ready: True. You can configure Backstage to surface these Kubernetes resource statuses directly on the service's catalog page. If a resource is failing, the developer sees a "Warning" status in the portal, which links them to the logs.

In clusters with >100 nodes, you'll notice that Crossplane's reconciliation loop can put significant pressure on the Kubernetes API server. I've seen cases where too many frequent updates to the status of 500+ cloud resources caused API latency. To mitigate this, tune the pollInterval in your Crossplane providers. Don't check every 60 seconds if a database is ready; 5 or 10 minutes is usually sufficient for infrastructure that takes 15 minutes to provision.

Another production gotcha is "orphaned resources." If a developer deletes the manifest from Git, ArgoCD deletes the Claim from Kubernetes, and Crossplane deletes the RDS instance. This is great for dev environments but catastrophic for production. You must implement a "deletion policy" in your Compositions. Set deletionPolicy: Orphan for production workloads. This ensures that if the YAML is accidentally deleted, the actual cloud resource remains intact.

Best Practices for Platform Engineering

Implementing an IDP is more of an organizational challenge than a technical one. If you build a perfect platform that no one uses, you've failed. Follow these principles to ensure adoption:

Start with the "Golden Path": Do not try to automate every possible cloud resource on day one. Identify the three most requested resources (for example, S3 buckets, Postgres DBs, and Redis caches) and build high-quality templates for those. This provides immediate value and builds trust.
Enforce Governance via Compositions: Use Crossplane Compositions to bake in security. Ensure every S3 bucket is encrypted and every RDS instance is in a private subnet by default. The developer shouldn't even see the "Encryption" checkbox; it should be mandatory and invisible.
Treat your IDP as a Product: Your developers are your customers. Conduct user interviews to find where the friction is. If they find the Backstage form too long, simplify it. If they need more visibility into costs, integrate a cost-tracking plugin.
Implement Strong RBAC: Use Kubernetes namespaces to isolate claims. Ensure that a developer in the team-a namespace cannot modify a PostgresDatabase claim in the team-b namespace. Use a tool like Kyverno to enforce these boundaries.
Version your Compositions: When you update a Composition (for example, upgrading the RDS instance class), don't just push it to production. Version your XRDs and Compositions so you can migrate services gradually rather than forcing a global update.

FAQ

How does this approach differ from using Terraform with a CI/CD pipeline?
Traditional Terraform requires a "push" model where a pipeline runs terraform apply. This often leads to state locking issues and configuration drift. The Backstage + Crossplane approach uses a "pull" model (Control Plane). Crossplane constantly monitors the state of the cloud and automatically corrects drift without needing a manual pipeline trigger.

Does this mean I have to migrate all my existing Terraform code to Crossplane?
No. You can run them side-by-side. Use Crossplane for new, self-service workloads while keeping your core networking and foundation (VPCs, IAM roles) in Terraform. You can even use the Terraform provider for Crossplane to manage existing Terraform modules through the Kubernetes API.

What happens if the cloud provider API is down during provisioning?
Crossplane employs an exponential back-off strategy. If the AWS API returns a 500 error, Crossplane will keep retrying the request. The Kubernetes resource will stay in a Synced: False state. Because you have a GitOps audit trail, you can easily see which resources are stuck.

Is Backstage overkill for small teams?
If you have fewer than five developers, a simple README and a set of shared Terraform modules might suffice. However, once you hit a scale where the platform team becomes a bottleneck for "simple" requests, the investment in Backstage pays off by eliminating the ticket queue.

Conclusion

Combining Backstage and Crossplane allows you to move from a culture of "ticket-based infrastructure" to true self-service. By using Backstage as the user interface and Crossplane as the control plane, you create a system where developers can provision production-ready resources in minutes, not days. This doesn't just speed up delivery; it allows your platform engineers to stop performing repetitive manual tasks and start focusing on high-value architectural improvements.

To get started, your first actionable step is to install Crossplane v1.14.x on a development cluster and create your first CompositeResourceDefinition for a simple resource, like an S3 bucket. Once the API is working, set up a basic Backstage instance and create a software template that outputs the YAML required by that XRD. Start small, validate the "Golden Path" with one team, and then scale the platform to the rest of your organization.

Essential kubectl Commands Cheat Sheet

DevOps Start — Tue, 14 Apr 2026 15:36:24 +0000

Stop memorizing every flag! I've put together a handy kubectl cheat sheet for managing pods, deployments, and debugging. Originally published on devopsstart.com.

Pod Management

Command	Description
`kubectl get pods`	List all pods in current namespace
`kubectl get pods -A`	List pods across all namespaces
`kubectl describe pod <name>`	Show detailed pod information
`kubectl delete pod <name>`	Delete a specific pod
`kubectl logs <pod>`	View pod logs
`kubectl logs <pod> -f`	Stream pod logs
`kubectl exec -it <pod> -- sh`	Open shell in pod

Deployments

Command	Description
`kubectl get deployments`	List all deployments
`kubectl scale deploy <name> --replicas=3`	Scale a deployment
`kubectl rollout status deploy/<name>`	Check rollout status
`kubectl rollout undo deploy/<name>`	Rollback a deployment
`kubectl set image deploy/<name> <container>=<image>`	Update container image

Services and Networking

Command	Description
`kubectl get svc`	List all services
`kubectl get ingress`	List all ingress resources
`kubectl port-forward svc/<name> 8080:80`	Forward local port to service
`kubectl get endpoints`	List service endpoints

Debugging

Command	Description
`kubectl get events --sort-by=.metadata.creationTimestamp`	View cluster events
`kubectl top pods`	Show pod resource usage
`kubectl top nodes`	Show node resource usage
`kubectl logs <pod> --previous`	View logs from crashed container
`kubectl describe node <name>`	Check node conditions

Context and Config

Command	Description
`kubectl config get-contexts`	List all contexts
`kubectl config use-context <name>`	Switch context
`kubectl config current-context`	Show current context
`kubectl get ns`	List namespaces
`kubectl config set-context --current --namespace=<ns>`	Set default namespace

Debug Kubernetes CrashLoopBackOff in 30 Seconds

DevOps Start — Tue, 14 Apr 2026 15:31:20 +0000

Struggling with a pod stuck in CrashLoopBackOff? This quick guide, originally published on devopsstart.com, shows you the exact commands to diagnose the root cause in seconds.

The Problem

Your pod is stuck in CrashLoopBackOff and you need to find out why — fast.

The Solution

kubectl logs <pod-name> --previous

The --previous flag shows logs from the last crashed container instance. This is the single most useful flag for debugging crash loops.

Combine with describe for the full picture

kubectl describe pod <pod-name> | grep -A 5 "Last State"

This shows the exit code and reason for the last termination:

Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137

Common Exit Codes

Code	Meaning	Fix
1	Application error	Check app logs
137	OOMKilled	Increase memory limits
139	Segfault	Check binary compatibility
143	SIGTERM	Graceful shutdown issue

Why It Works

Kubernetes keeps logs from the previous container instance even after it crashes. Without --previous, you'd only see logs from the current (possibly empty) instance that hasn't had time to produce output before crashing again.

Rapid Rollback: `kubectl set image` for Urgent Fixes

DevOps Start — Tue, 14 Apr 2026 15:26:17 +0000

Originally published on devopsstart.com. When production breaks, every second counts—here is how to use kubectl set image for a precise and rapid rollback.

Introduction

You've just deployed a new container image to production, and almost immediately, monitoring alerts start screaming. Latency is spiking, error rates are through the roof, and your customers are experiencing service degradation. In these high-pressure moments, a fast, reliable rollback mechanism is critical. While Kubernetes offers robust rollout and rollback capabilities via kubectl rollout undo, there are specific scenarios where kubectl set image can provide a quicker, more direct path to recovery, especially when you know exactly which image version you need to revert to.

This tip focuses on leveraging kubectl set image for urgent rollbacks. You'll learn when this command is most effective, how to accurately identify the correct previous image tag, and how to execute the command to quickly stabilize your application.

Understanding `kubectl set image`

The kubectl set image command is primarily designed to atomically update the image of one or more specific containers within a Kubernetes resource. It typically targets Deployments, StatefulSets, DaemonSets, or ReplicationControllers. When executed, it modifies the resource's Pod template to point to the new image tag, which then triggers a new rolling update.

While kubectl set image is frequently used for forward deployments (e.g., updating v1.1.9 to v1.2.0), its direct nature makes it exceptionally well-suited for rapid rollbacks. When you specify a previous, stable image, Kubernetes initiates a new rollout toward that desired state. This behavior differentiates it from kubectl rollout undo, which inherently steps back through the deployment's recorded history, revision by revision.

Here’s a common example of how kubectl set image is used to update an image:

kubectl set image deployment/my-app my-container=my-registry/my-app:v1.2.0 -n production

Expected output:

deployment.apps/my-app image updated

This command updates my-container within the my-app deployment in the production namespace to use the v1.2.0 image from my-registry.

The Urgent Rollback Scenario

Consider this scenario: your my-app:v1.2.0 release introduced a critical bug that bypassed your staging environment checks. You pushed it to production an hour ago, and now, critical alerts are firing, indicating significant application failures. You need to revert to the last known good image, let's say my-app:v1.1.9, immediately.

Why might kubectl set image be preferred over kubectl rollout undo in such a situation?

Directness and Precision: If you know the exact, stable image tag to which you need to revert, kubectl set image offers an explicit and precise command. This avoids ambiguity and ensures you land on the intended stable state directly.
Bypassing Unhealthy Revisions: If multiple faulty deployments occurred after your last stable one (e.g., you tried v1.2.0, then v1.2.1-hotfix, both failed), kubectl rollout undo would sequentially step back through these potentially problematic revisions. kubectl set image allows you to jump directly to the known good v1.1.9 without traversing the unstable intermediate states.
Forced Redeploy (Edge Cases): In rare cases, even if an image tag is theoretically the same, you might want to force Kubernetes to re-pull container images and redeploy pods due to local caching issues or other inconsistencies. Re-setting the image explicitly with kubectl set image can achieve this, ensuring fresh pods are created. For more on debugging common Kubernetes issues, refer to our article on Troubleshooting CrashLoopBackOff in Kubernetes.

Identifying the Previous Image Tag

The critical first step for a kubectl set image rollback is accurately identifying the last known good image tag. You can achieve this by inspecting your deployment's revision history:

Check Rollout History:
This command provides a concise summary of your deployment's revision history, showing the changes made at each step.
```
kubectl rollout history deployment/my-app -n production
```


shell

    **Expected output:**


```bash
    deployment.apps/my-app 
    REVISION  CHANGE-CAUSE
    1         <none>
    2         my-container: my-registry/my-app:v1.1.8
    3         my-container: my-registry/my-app:v1.1.9
    4         my-container: my-registry/my-app:v1.2.0

From this output, if `v1.2.0` (revision 4) is currently causing issues, then `v1.1.9` (revision 3) is your immediate target for rollback. Note that `CHANGE-CAUSE` may also contain details if `--record` was used during deployment.

Describe a Specific Revision (Optional Verification):
To be absolutely certain about the container images used in a particular revision, you can describe it in detail. This is a good verification step before initiating a rollback.
```
kubectl rollout history deployment/my-app --revision=3 -n production
```


shell

    **Expected (truncated) output:**


```bash
    deployment.apps/my-app with revision 3
    Pod Template:
      Labels:       app=my-app
                    pod-template-hash=54c9c76...
      Containers:
        my-container:
          Image:        my-registry/my-app:v1.1.9
          Port:         8080/TCP
          Environment:  <none>
    ...

This confirms that `my-registry/my-app:v1.1.9` was indeed the image used for revision 3, making it a reliable rollback target.

Executing the `kubectl set image` Rollback

Once you have identified the precise desired image tag (e.g., my-registry/my-app:v1.1.9 in our example), executing the rollback is straightforward and immediate:

kubectl set image deployment/my-app my-container=my-registry/my-app:v1.1.9 -n production

Expected output:

deployment.apps/my-app image updated

Upon execution, Kubernetes will immediately initiate a new rolling update. It will begin replacing the currently failing v1.2.0 pods with new ones running the specified stable v1.1.9 image.

You can monitor the progress of this new rollout using the following command:

kubectl rollout status deployment/my-app -n production

Expected output during rollout:

Waiting for deployment "my-app" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "my-app" rollout to finish: 1 old replicas are pending termination...
deployment "my-app" successfully rolled out

Once the rollout is complete, your application should be consistently running the stable v1.1.9 image, and your monitoring alerts should ideally begin to subside as service is restored.

Important Considerations

Rollback Strategy Impact: This kubectl set image method performs a rolling update. It's crucial that your application is designed to handle a brief period where both the old (problematic) and new (stable) versions of pods are running concurrently. This typically means ensuring backward and forward compatibility for APIs and data schemas.
Image Immutability: Always strive to use immutable image tags (e.g., v1.1.9, v1.2.0, sha256:abcdef...) rather than mutable tags like latest. Immutable tags guarantee that a specific tag always refers to the exact same image content, which is fundamental for reliable and reproducible rollbacks.
Auditing and History: Using kubectl set image creates a new revision in the deployment's history. This automatically ensures that your rollback action is recorded, providing a clear audit trail of changes made to your deployment.
Stateful Workloads: For StatefulSets, exercising caution when changing image versions is paramount. If a new image version introduces changes that affect persistent storage or state, a simple image rollback might not fully resolve database schema migrations or data portability issues. Always understand the data implications.

Conclusion

When a problematic image release throws production into disarray, reaction time is paramount. While kubectl rollout undo is a valuable tool, kubectl set image provides a direct, efficient, and precise alternative for reverting to a specific, known-good image. This capability can significantly reduce Mean Time To Recovery (MTTR) by allowing you to bypass potentially multiple failing revisions and jump straight to stability. By understanding your deployment history and precisely targeting the last stable

How to Set Up Argo CD GitOps for Kubernetes Automation

DevOps Start — Tue, 14 Apr 2026 15:21:13 +0000

Stop relying on manual kubectl applies and start treating your cluster as code. This comprehensive guide, originally published on devopsstart.com, walks you through setting up Argo CD for true GitOps automation.

Introduction

If you are still running kubectl apply -f manifests/ from your local machine or a Jenkins pipeline, you are operating in a "Push" model. In this model, your CI tool needs high-privileged credentials to your cluster, and you have no guarantee that what is actually running in production matches what is in your Git repository. One rogue developer running a manual kubectl edit can create a "configuration drift" that haunts you for months.

This is where GitOps comes in. GitOps is a paradigm where Git is the single source of truth for your infrastructure and application state. Instead of pushing changes to the cluster, a controller inside the cluster constantly monitors your Git repo and "pulls" the state to match.

In this tutorial (Part 1 of our series), we will move from imperative deployments to declarative continuous delivery using Argo CD v2.11.0. You'll learn how to install Argo CD, connect your repositories, handle configuration drift, and scale your deployments using ApplicationSets. By the end, you'll have a production-ready GitOps engine that ensures your cluster is always in the desired state. For a deeper dive into how this compares to other tools, check out our guide on /blog/argo-cd-vs-flux-a-guide-for-multi-cluster-gitops.

Prerequisites

Before we start, you need a working Kubernetes cluster. This can be a managed service like EKS, GKE, or AKS, or a local setup like Kind or Minikube. For this tutorial, we assume you are using Kubernetes v1.30 or newer.

You will need the following tools installed on your local workstation:

kubectl v1.30+: The standard Kubernetes CLI. Ensure your context is set to the correct cluster.
Git v2.40+: Required for managing the manifests that Argo CD will track.
A GitHub or GitLab account: You need a repository to store your Kubernetes YAML manifests.
Basic YAML knowledge: You should know how to write a basic Deployment and Service manifest. If you are totally new to this, refer to our /blog/kubernetes-for-beginners-deploy-your-first-application.

You don't need to install the Argo CD CLI for the basic setup, as we will use the Web UI and kubectl for most operations, but having it installed is helpful for advanced automation.

Overview

The goal of this tutorial is to build a fully automated deployment pipeline where a Git commit is the only trigger needed to update your application. We aren't just deploying a "Hello World" app; we are building a scalable architecture.

Here is the architecture we will implement:

The Manifest Repo: A dedicated Git repository containing the desired state of your cluster (YAML files).
Argo CD Controller: Installed in the argocd namespace, acting as the GitOps operator.
The Application CRD: A custom resource that tells Argo CD: "Watch this folder in Git and make sure it exists in this namespace in the cluster."
Automated Sync: A policy that automatically corrects any manual changes (drift) made to the cluster.
ApplicationSets: A template-based approach to deploy the same application across multiple namespaces (e.g., dev, staging, prod) without duplicating YAML files.

By moving to this "Pull" model, you eliminate the need to store Kubeconfigs in your CI tool (like GitHub Actions or GitLab CI), which reduces the attack surface of your infrastructure by removing high-privileged secrets from external runners.

Step 1: Installing Argo CD

Argo CD is installed as a set of deployments and services within your cluster. We will use the official manifests provided by the Argo project.

First, create a dedicated namespace for Argo CD to keep the installation isolated.

kubectl create namespace argocd

Now, apply the installation manifest. We will use the stable release manifest for v2.11.0.

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.11.0/manifests/install.yaml

Wait for all pods to reach the Running state. You can monitor the progress with this command:

kubectl get pods -n argocd

The output should look like this:

NAME                                                 READY   STATUS    RESTARTS   AGE
argocd-server-7d5f8f8f5-abc12                       1/1     Running   0          2m
argocd-repo-server-5f4d7e9-def34                     1/1     Running   0          2m
argocd-application-controller-f7e8d9-ghi56           1/1     Running   0          2m
argocd-redis-7c8b9a0-jkl78                           1/1     Running   0          2m
argocd-notifications-controller-h9i0j1-mno90         1/1     Running   0          2m

If any pods stay in Pending or CrashLoopBackOff, you can diagnose the issue using our guide on /troubleshooting/crashloopbackoff-kubernetes.

Step 2: Initial Access and Authentication

By default, the Argo CD API server is not exposed to the public internet. For this tutorial, we will use port-forwarding to access the UI.

Start the port-forward in a separate terminal window:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Now, open your browser and go to https://localhost:8080. You will see a login screen. The default username is admin. The password, however, is automatically generated and stored in a Kubernetes secret.

Run the following command to retrieve the initial admin password:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

The output will be a plain-text string, for example: xYz123AbC456DefG. Copy this password and use it to log in to the UI.

Once you log in, change the admin password under the "User Management" settings. For production environments, avoid using the initial admin account and instead integrate with an OIDC provider like Okta or GitHub. You can find more details in the official Argo CD documentation.

Step 3: Connecting Your Git Repository

Argo CD needs permission to read your manifests. If your repository is public, you can just provide the URL. If it is private, you need to provide SSH keys or HTTPS credentials.

Let's assume you have a private GitHub repository located at git@github.com:your-org/gitops-manifests.git.

Log in to the Argo CD UI.
Go to Settings $\rightarrow$ Repositories.
Click Connect Repo.
Select via SSH.
Enter the Repository URL: git@github.com:your-org/gitops-manifests.git.
Paste your private SSH key (the one that has read access to the repo).
Click Connect.

If the connection is successful, the status will change to Successful. If you see Failed, ensure your SSH key is correct and that the Argo CD pod has outbound network access to GitHub.

Step 4: Creating Your First GitOps Application

In Argo CD, an "Application" is a Custom Resource (CRD) that defines the link between a source (Git) and a destination (Cluster).

We will create a simple application that deploys a guestbook app. First, ensure your Git repo has a folder named guestbook containing a deployment.yaml and a service.yaml.

Using YAML is the "GitOps way" because you can store the Application definition itself in Git (the App-of-Apps pattern). Save the following as guestbook-app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook-gitops
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'git@github.com:your-org/gitops-manifests.git'
    targetRevision: HEAD
    path: guestbook
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: guestbook-demo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Component Breakdown

source: This tells Argo CD where to look. targetRevision: HEAD tracks the latest commit on the default branch.
destination: https://kubernetes.default.svc refers to the cluster where Argo CD is currently installed.
syncPolicy:
- automated: Argo CD will automatically apply changes from Git to the cluster.
- prune: true: If you delete a file from Git, Argo CD will delete the corresponding resource from Kubernetes.
- selfHeal: true: If someone manually edits a resource in the cluster, Argo CD will instantly overwrite it with the Git version.
- CreateNamespace=true: Ensures the guestbook-demo namespace is created if it doesn't exist.

Apply this manifest:

kubectl apply -f guestbook-app.yaml

Now, go to the Argo CD UI. You will see the guestbook-gitops application. Initially, it will be "OutOfSync" while it calculates the difference, then it will transition to "Synced" and "Healthy" as it creates the pods and services.

Step 5: Handling Configuration Drift

Configuration drift occurs when the actual state of the cluster deviates from the desired state defined in Git. This usually happens when a developer uses kubectl edit to fix a production bug quickly but forgets to update the Git repo.

Let's simulate this. We will manually scale our deployment to 5 replicas using the CLI.

kubectl scale deployment guestbook --replicas=5 -n guestbook-demo

If you check the Argo CD UI now, the application status has changed from Synced to OutOfSync. The UI will highlight the exact difference in yellow: "Desired: 3 replicas, Actual: 5 replicas."

Because we enabled selfHeal: true, you won't have to do anything. Within a few seconds, Argo CD will detect the drift and automatically scale the deployment back down to 3 replicas to match Git.

If selfHeal were disabled, the application would stay OutOfSync. You would then have two choices:

Sync: Click the "Sync" button in the UI to force the cluster to match Git.
Update Git: Change the replica count in your Git repo to 5, commit, and push. Argo CD would then see the new desired state and update the cluster.

This mechanism ensures that your Git history is an audit log of every change ever made to your environment.

Step 6: Scaling with ApplicationSets

Creating one Application manifest for one app is easy. But if you have 50 microservices across dev, staging, and prod clusters, creating 150 Application manifests is a maintenance nightmare.

ApplicationSets allow you to use a template to generate multiple Applications automatically. They use "generators" to discover targets.

Let's use a List Generator to deploy the same guestbook app into three different namespaces. Save this as guestbook-appset.yaml:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook-environments
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: engineering-dev
            namespace: guestbook-dev
          - cluster: engineering-staging
            namespace: guestbook-staging
          - cluster: engineering-prod
            namespace: guestbook-prod
  template:
    metadata:
      name: '{{cluster}}-guestbook'
    spec:
      project: default
      source:
        repoURL: 'git@github.com:your-org/gitops-manifests.git'
        targetRevision: HEAD
        path: guestbook
      destination:
        server: 'https://kubernetes.default.svc'
        namespace: '{{namespace}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Apply the ApplicationSet:

kubectl apply -f guestbook-appset.yaml

Argo CD iterates through the list and creates three separate Application resources: engineering-dev-guestbook, engineering-staging-guestbook, and engineering-prod-guestbook.

If you need to add a new environment (e.g., qa), you simply add one line to the elements list in the ApplicationSet and commit. For teams managing huge fleets of clusters, the Cluster Generator is even more powerful; it can automatically detect every cluster registered in Argo CD and deploy a "base" set of tools to all of them without any manual listing.

Step 7: Integrating the Full CI/CD Pipeline

Now that the "CD" (Continuous Delivery) part is handled by Argo CD, how does the "CI" (Continuous Integration) part fit in?

A common mistake is letting the CI tool (GitHub Actions, Jenkins, CircleCI) call kubectl apply. This breaks the GitOps model. Instead, the CI tool should only be responsible for updating the manifest repository.

Here is the professional workflow:

Developer Pushes Code: A developer pushes a change to the application source code.
CI Pipeline Runs: GitHub Actions triggers a build, runs tests, and builds a Docker image.
Image Push: The CI tool pushes the image to a registry (e.g., Amazon ECR) with a unique tag (the Git SHA).
Manifest Update: The CI tool clones the gitops-manifests repo and updates the image tag in the deployment.yaml file.
Git Commit: The CI tool commits and pushes the change back to the manifest repo.
Argo CD Pull: Argo CD detects the commit in the manifest repo and pulls the change into the cluster.

Example GitHub Action snippet for manifest update

- name: Update Kubernetes image tag
  run: |
    git clone https://x-token-auth:${{ secrets.GITOPS_TOKEN }}@github.com/your-org/gitops-manifests.git
    cd gitops-manifests
    sed -i "s|image: my-app:.*|image: my-app:${{ github.sha }}|g" guestbook/deployment.yaml
    git config user.name "GitHub Action"
    git config user.email "action@github.com"
    git add .
    git commit -m "Update guestbook image to ${{ github.sha }}"
    git push

This separation of concerns is critical. The CI tool has no access to the cluster; it only has access to a Git repository. If your CI tool is compromised, the attacker cannot delete your production pods; they can only propose changes to Git, which can be blocked by a Pull Request review.

Troubleshooting

1. Application stuck in "Progressing" status

If your application is "Synced" but stays in "Progressing", the pods are likely failing to start. Argo CD is waiting for the Kubernetes health check to return Healthy.

Check the pod events:

kubectl get pods -n guestbook-demo
kubectl describe pod <pod-name> -n guestbook-demo

Look for ImagePullBackOff or CrashLoopBackOff. If you see the latter, use the tips in /tips/debug-crashloopbackoff to find the root cause.

2. Repository Connection Failed

If Argo CD cannot connect to your Git repo, check the argocd-repo-server logs:

kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server

Common causes include:

Incorrect SSH private key.
Firewall rules blocking port 22 (SSH) or 443 (HTTPS) from the cluster to GitHub.
Using a GitHub Deploy Key that doesn't have read access to the repository.

3. "OutOfSync" loop

Sometimes an application flips between Synced and OutOfSync rapidly. This is often caused by a conflict between Argo CD and another controller (like a Horizontal Pod Autoscaler).

If HPA is changing the replica count and Argo CD is trying to force it back to the Git value, they will fight forever. To fix this, ignore the replicas field in the Application spec:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

FAQ

Q: Can Argo CD manage Helm charts?
A: Yes. Argo CD natively supports Helm. You can point the source to a Helm repository or a folder containing a Chart.yaml. You can also provide a values.yaml file in Git to override default settings.

Q: What is the "App-of-Apps" pattern?
A: This is a pattern where you create one "Root" Argo CD Application that points to a folder containing other Application manifests. This allows you to manage your entire cluster state (including other apps) using a single GitOps entry point.

Q: Does Argo CD support multi-cluster management?
A: Yes. You can add external clusters to Argo CD via the CLI or UI. Once added, you can set the destination.server in your Application manifest to the API server URL of the remote cluster.

Conclusion

You have now transitioned from manual kubectl deployments to a professional GitOps workflow. We have installed Argo CD v2.11.0, connected a private repository, and deployed an application using the declarative model. By implementing selfHeal, you've ensured that your cluster is resilient to manual configuration drift. Furthermore, by using ApplicationSets, you've built a foundation that can scale from one application to hundreds across multiple environments.

The key takeaway is the shift in trust. You no longer trust the state of the cluster; you trust the state of Git. This makes your deployments repeatable, auditable, and significantly more secure.

Actionable next steps:

Migrate one existing production service to Argo CD.
Set up a "Management" repo that uses the App-of-Apps pattern to manage all your other Application manifests.
Implement a PR-based workflow where no one is allowed to push directly to the main branch of your manifest repo.

In Part 2 of this series, we will cover advanced Argo CD features, including Blue/Green and Canary deployments using Argo Rollouts, and how to integrate Prometheus metrics to trigger automatic rollbacks if a new release increases your error rate.

How to Configure Advanced Argo CD Sync Policies for GitOps

DevOps Start — Tue, 14 Apr 2026 15:16:09 +0000

Want to move beyond basic GitOps? I've put together a deep dive on mastering Argo CD sync policies, originally published on devopsstart.com.

Prerequisites

Before diving into advanced sync policies, you need a functioning Kubernetes cluster and a baseline Argo CD installation. This tutorial assumes you've already moved past the "Hello World" phase of GitOps. If you haven't set up your initial environment yet, follow the guide on /tutorials/how-to-set-up-argo-cd-gitops-for-kubernetes-automation to get the controller running.

To follow the examples in this guide, ensure the following tools are installed on your local machine:

Kubernetes Cluster: v1.28 or newer.
kubectl: v1.28 or newer, configured to communicate with your cluster.
Argo CD CLI: v2.10.0 or newer. This is essential for performing manual rollbacks and interacting with the API without the GUI.
Git: A repository (GitHub, GitLab or Bitbucket) containing your Kubernetes manifests.
A Sample Application: A deployment consisting of at least one Deployment, one Service and one ConfigMap.

You should have a basic understanding of the Application CRD (Custom Resource Definition) and how Argo CD tracks the state between your Git repository (the desired state) and your cluster (the live state). If you are unsure how to structure your Git folders, refer to /tutorials/how-to-set-up-argo-cd-gitops-for-kubernetes-automation.

Overview

Most teams start with Argo CD using "Manual Sync." You push code to Git, see the "Out of Sync" yellow badge in the UI and click the "Sync" button. While this feels safe, it's not production-grade. In large-scale environments, manual syncing creates a bottleneck and leads to configuration drift, where the cluster state diverges from Git for hours because a manual trigger was missed.

Simply turning on "Automatic Sync" can be dangerous. By default, Argo CD ensures that what is in Git is present in the cluster, but it won't necessarily remove what is not in Git. This leads to orphaned resources (leftover services or secrets) that can cause naming conflicts or security holes.

In this tutorial, we will build a production-ready synchronization strategy. You will learn how to implement automated pruning to keep your cluster clean, configure self-healing to prevent manual "hot-fixes" from persisting and manage complex deployment orders using sync waves. We will also tackle the "Day 2" problem of rollbacks: deciding when to revert a Git commit versus using the Argo CD rollback feature.

By the end of this guide, you'll have a robust GitOps pipeline that handles infrastructure lifecycle management automatically, reduces human error during deployments and provides a clear path for disaster recovery. You can find more details on the core architecture in the official Argo CD Documentation.

Step 1: Implementing Automated Pruning and Self-Healing

The first step toward production-grade GitOps is eliminating manual intervention. Many operators avoid prune: true fearing the accidental deletion of production resources. However, without pruning, your cluster becomes a graveyard of old ConfigMaps and abandoned Services.

Understanding Pruning and Self-Healing

Pruning is the process where Argo CD identifies resources that exist in the cluster (and are managed by the app) but are no longer present in the Git repository. If pruning is disabled, deleting a file in Git does nothing to the cluster.

Self-healing goes a step further. If a developer uses kubectl edit to change a replica count or an environment variable directly in the cluster, Argo CD detects the drift and immediately overwrites those changes with the state defined in Git.

Configuration

To enable these, modify the syncPolicy section of your Application manifest.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook-production
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://github.com/argoproj/argocd-example-apps.git'
    targetRevision: HEAD
    path: guestbook
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: guestbook
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Apply this configuration using kubectl:

kubectl apply -f application.yaml

Testing the Policy

To verify pruning, delete a resource from your Git repository (for example, a Service manifest) and push the change.

git rm manifests/service.yaml
git commit -m "Remove legacy service"
git push origin main

Wait for the next sync cycle (usually 3 minutes by default, or instantly if you have a webhook configured), then check your cluster:

kubectl get svc -n guestbook
# Expected output: Error from server (NotFound)

Now, test self-healing. Try to manually scale your deployment:

kubectl scale deployment guestbook-ui --replicas=10 -n guestbook

Run kubectl get pods -n guestbook. You'll notice the pods scale up for a moment, but within 60 to 120 seconds, Argo CD will detect the drift and scale them back down to the number specified in Git.

Step 2: Mastering Advanced Sync Options

Standard syncing works for 90% of resources, but Kubernetes has immutable fields. For example, if you try to change the selector of a Service or certain fields in a Job, the Kubernetes API rejects the update with a 422 Unprocessable Entity error. Argo CD will remain in a "Sync Failed" state indefinitely.

Using Replace=true

The Replace=true option tells Argo CD to use kubectl replace or kubectl create instead of kubectl apply. This effectively deletes and recreates the resource if an update fails due to immutable fields.

Add this to your syncOptions:

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - Replace=true
      - SkipDryRunOnMissingResource=true

SkipDryRunOnMissingResource=true is particularly useful when dealing with complex CRDs. Sometimes the dry-run validation fails because a dependent resource doesn't exist yet, even though the actual application would succeed.

ApplicationSet Level Policies

If you manage 50 clusters using an ApplicationSet, you don't want to define these policies 50 times. Define the syncPolicy within the template section of the ApplicationSet.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-config
spec:
  generators:
    - list:
        elements:
          - cluster: engineering-dev
            url: https://kubernetes.default.svc
          - cluster: engineering-prod
            url: https://prod-cluster.example.com
  template:
    metadata:
      name: '{{cluster}}-guestbook'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/argoproj/argocd-example-apps.git'
        targetRevision: HEAD
        path: guestbook
      destination:
        server: '{{url}}'
        namespace: guestbook
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Step 3: Implementing Sync Waves and Hooks

In production, you cannot deploy everything simultaneously. You might need a database schema migration to finish before the API server starts, or a smoke test to pass before the LoadBalancer switches traffic.

Sync Waves

Sync waves allow you to assign an order to resources. Argo CD applies resources in increasing order of their wave number. Resources with the same wave are applied concurrently.

Add the annotation argocd.argoproj.io/sync-wave to your manifests.

Database Migration (Wave 1):

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: migration-tool:v1.2.0
      restartPolicy: OnFailure

Application Deployment (Wave 2):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  # Deployment spec here

Cache Warmup (Wave 3):

apiVersion: batch/v1
kind: Job
metadata:
  name: cache-warmup
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  # Job spec here

Argo CD waits for the Wave 1 Job to reach a "Healthy" state before attempting to create the Wave 2 Deployment.

Sync Hooks

Hooks are used for transient tasks rather than permanent resources.

PreSync: Runs before the sync starts. Ideal for backups.
Sync: Runs during the sync.
PostSync: Runs after the sync completes. Ideal for notifications or integration tests.

Example of a PreSync backup hook:

apiVersion: batch/v1
kind: Job
metadata:
  name: pre-sync-backup
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: backup
          image: backup-util:latest
      restartPolicy: OnFailure

The HookSucceeded policy ensures the Job object is deleted from the cluster once it completes successfully, preventing the buildup of thousands of finished Job objects.

Step 4: Designing Robust Rollback Strategies

When a production deployment fails, the pressure to recover "right now" often leads to a conflict between "Pure GitOps" and "Fast Recovery."

Strategy A: The Git-based Rollback (Pure GitOps)

In this approach, you never use the Argo CD UI for rollbacks. You use git revert.

Pros:

Perfect audit trail.
Zero drift between Git and Cluster.
Works across multiple clusters simultaneously.

Cons:

Slower recovery time. You must commit, push and wait for the sync cycle.

Execution:

git log --oneline
# a1b2c3d (HEAD) Update image to v2.1.0 (BROKEN)
# e5f6g7h Update image to v2.0.0 (STABLE)

git revert a1b2c3d
git push origin main

Strategy B: The Argo CD UI/CLI Rollback (Emergency Fast-Track)

Argo CD allows you to rollback to a previous successful revision of the application. This is an immediate operation that bypasses Git.

Execution using CLI:

argocd app rollback guestbook-production 12

The Danger Zone: If automated: selfHeal: true is enabled, a manual rollback will be immediately overwritten. Argo CD will see the cluster is running v2.0.0 (due to the rollback) while Git still says v2.1.0. Because self-healing is on, it will "fix" the cluster by re-deploying the broken v2.1.0.

The Decision Matrix

Follow these rules for professional rollback management:

For Non-Critical Bugs: Use git revert. It is the only way to ensure the environment remains reproducible.
For Critical Outages (P0):

Step 1: Disable Auto-Sync in the UI or CLI.
Step 2: Perform the Argo CD Rollback to a known good revision.
Step 3: Fix the code in Git.
Step 4: Update Git to the fixed version.
Step 5: Re-enable Auto-Sync.

If you encounter constant Pod failures during these transitions, you might be facing a /troubleshooting/crashloopbackoff-kubernetes scenario, which requires log analysis before deciding on a rollback strategy.

Step 5: Implementing Custom Health Checks

Argo CD knows how to check the health of standard resources. However, if you use Custom Resource Definitions (CRDs) from an operator (like Prometheus or Istio), Argo CD only knows if the resource was created. It doesn't know if the operator actually succeeded in deploying the underlying components.

This means a Sync Wave might move to Wave 2 even if the Wave 1 CRD is still in a "Pending" or "Error" state.

Defining a Lua Health Check

Argo CD allows you to define health checks using Lua scripts in the argocd-cm ConfigMap in the argocd namespace.

Assume you have a custom resource called DatabaseInstance that has a status.phase field.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.apps.example.com/DatabaseInstance: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.phase == 'Ready' then
        hs.status = 'Healthy'
        hs.message = 'Database is ready'
        return hs
      end
      if obj.status.phase == 'Failed' then
        hs.status = 'Degraded'
        hs.message = 'Database failed to provision'
        return hs
      end
    end
    hs.status = 'Progressing'
    hs.message = 'Waiting for database to be ready'
    return hs

Apply the change and restart the argocd-application-controller:

kubectl apply -f argocd-cm.yaml
kubectl rollout restart deployment argocd-application-controller -n argocd

Now, Argo CD will wait for the DatabaseInstance to reach the Ready phase before marking the resource as Healthy.

Step 6: Managing Sync Windows and Maintenance Periods

In enterprise environments, automated deployments are often prohibited during "Freeze Periods" (e.g., Black Friday). You still want GitOps to track changes, but you don't want them applied to the cluster.

Argo CD doesn't have a built-in calendar, but you can implement this using labels and automation.

The Label-Based Freeze Approach

Add a label sync-window: frozen to your application.

kubectl label app guestbook-production sync-window=frozen

Create a simple automation (via GitHub Action or CronJob) that toggles the automated sync policy.

The "Freeze" Script:

# Disable auto-sync during freeze
argocd app set guestbook-production --sync-policy manual

The "Unfreeze" Script:

# Enable auto-sync after freeze
argocd app set guestbook-production --sync-policy automated

For a more sophisticated approach, use an external controller that watches for these labels and modifies the Application spec. This ensures the cluster remains untouched until the window opens.

Troubleshooting

Issue 1: Resource "Flickering" (The Sync Loop)

A resource constantly switches between "Synced" and "Out of Sync." This typically happens when a controller (like an HPA or Service Mesh) modifies the resource after Argo CD applies it.

The Fix: Use ignoreDifferences to tell Argo CD to ignore fields managed by other controllers.

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

Issue 2: Pruning Deleted Critical Resources

You accidentally deleted a namespace or a critical Secret in Git, and Argo CD pruned it from the cluster, causing an outage.

The Fix: Use the prune safety override. Annotate specific resources to prevent them from being pruned, regardless of the application-level policy.

metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false

Issue 3: Sync Wave Hanging

A sync wave is stuck in "Progressing" and refuses to move to the next wave.

The Fix: Check the health of the resource in the current wave. If it's a Job, ensure it is actually completing. If you implemented a custom health check, ensure the Lua script isn't returning Progressing indefinitely due to a typo in the status field.

kubectl describe job db-migration -n guestbook

FAQ

Q: Does prune: true delete resources in namespaces not managed by Argo CD?
A: No. Argo CD only prunes resources that are tracked within the specific Application's scope and managed by that application.

Q: Can I have different sync waves for different clusters?
A: Yes. Since sync waves are defined as annotations on the manifests themselves, you can use Kustomize or Helm to apply different annotations based on the target environment (e.g., a longer warmup wave in production than in dev).

Q: What happens if a Sync Hook fails?
A: If a PreSync hook fails, Argo CD will stop the sync process and mark the application as "Degraded," preventing the deployment of potentially broken code.

Q: Is Replace=true safe for all resources?
A: Not always. Since it deletes and recreates the resource, any fields not defined in your Git manifest (like some dynamically assigned annotations or labels) will be lost. Use it only for resources with immutable fields.

Conclusion

Moving from manual synchronization to advanced sync policies separates a "demo" GitOps setup from a production-grade platform. By implementing automated pruning and self-healing, you eliminate configuration drift and ensure Git is the absolute source of truth. Sync waves and hooks bring the orchestration capabilities of traditional CI/CD pipelines into the declarative world of Kubernetes.

In this tutorial, we've covered:

Enabling prune and selfHeal to maintain cluster hygiene.
Using Replace=true to handle immutable Kubernetes fields.
Orchestrating complex deployments with Sync Waves and Hooks.
The critical distinction between Git reverts and Argo CD rollbacks.
Extending Argo CD's intelligence with custom Lua health checks.
Managing deployment freezes using sync windows.

Your next steps should be to audit your current Application manifests. Identify resources managed by other controllers and apply ignoreDifferences to stop sync flickering. Then, map out your application dependencies and assign sync waves to ensure your databases always precede your APIs.

This concludes our deep dive into Argo CD and our series on GitOps automation. For those looking to further their expertise in site reliability, our guide on /interview/senior-sre-interview-questions-answers-for-2026 provides insight into how these patterns are evaluated in professional settings.

DEV Community: DevOps Start

Secure Terraform PRs with an Architecture Firewall

Introduction

Why Manual PR Reviews Fail the Architecture Test

Implementing the Policy as Code Engine

Real-World Application: Preventing Production Catastrophes

Best Practices for Architecture Guardrails

FAQ

Does the architecture firewall replace the need for manual peer reviews?

Which tool should I choose: OPA, Checkov, or Sentinel?

How do I prevent the firewall from slowing down my deployment pipeline?

Can I use this to manage costs?

Conclusion

Local LLM for Log Analysis: Privacy-First Debugging with Ollama

Introduction

Why this take

The strongest counter-argument

Exceptions where cloud LLMs still win

Implementing the Privacy-First Stack

Installing the Engine

The Log Pipeline Architecture

Prompt Engineering for DevOps

Hardware Requirements and Performance

Semantic Anomaly Detection vs. Regex

Log Streamlining and Noise Reduction

How to Build AI Agents for Kubernetes Deployments

Introduction

Prerequisites

Overview

Step 1: Architecting the Agent Loop

Step 2: Implementing the Tooling Layer with MCP

Step 3: Configuring Least-Privilege RBAC

Step 4: Integrating K8sGPT for Diagnostic Skills

Step 5: Building the Resource Optimization Skill

Step 6: Automating GitOps with ArgoCD Integration

Step 7: Implementing Safety Rails and Human-in-the-Loop (HITL)

1. The Dry-Run Constraint

2. The Approval Gate (HITL)

3. Policy-as-Code (Kyverno/OPA)

Step 8: Testing and Validating Agent Performance

Unit Testing Tools

Scenario-Based Validation (Chaos Engineering)

Token Cost and Latency Tracking

Troubleshooting

The Agent "Loops" Infinitely

RBAC "Forbidden" Errors

Hallucinated CLI Flags

Context Window Overflow

Conclusion

How to Manage Multiple Azure Subscriptions in Terraform

How to Manage Multiple Azure Subscriptions in Terraform

Implementing Provider Aliases

Cross-Subscription Data Referencing

The Module Provider Gotcha

Best Practices for Naming and Scale

FAQ

Conclusion

GitHub Actions Security: How to Stop Secret Leaks in CI/CD

Introduction

The Anatomy of a Secret Leak: Why Your Logs Aren't Safe

Moving from Static Secrets to OIDC

Hardening the Supply Chain: The Danger of Mutable Tags

Defending Against "Pwn-Requests" and Fork Attacks

Best Practices for CI/CD Hardening

FAQ

Does GitHub really mask all my secrets in the logs?

Why is pull_request_target worse than pull_request?

Should I use OIDC for every single cloud provider?

Can I still use version tags like @v4 if I use a private runner?

Conclusion

Cursor vs Copilot vs Cody: Best AI Editor for DevOps

Introduction

The Context Problem: Why General AI Fails DevOps

Cursor: The AI-Native Powerhouse for IaC

Sourcegraph Cody: Mastering the Enterprise Monorepo

Comparing AI Performance on YAML and HCL

Best Practices for AI-Driven DevOps

FAQ

Which AI editor is the most secure for corporate code?

Can these tools actually replace writing Terraform by hand?

Why is `pull_request_target` worse than `pull_request`?

Can I still use version tags like `@v4` if I use a private runner?

Understanding `kubectl set image`

Executing the `kubectl set image` Rollback