DEV Community: varun varde

Team Topologies for DevOps: A Practical Implementation Guide

varun varde — Thu, 21 May 2026 11:35:55 +0000

Most engineering organisations do not fail because their developers are untalented.

They fail because their communication structures, ownership boundaries, and operational dependencies create friction that compounds over time.

A deployment takes three weeks because four teams must approve it. A platform team becomes a ticket queue instead of a product team. Stream-aligned teams spend more time negotiating dependencies than shipping software. Cognitive overload silently accumulates until incident frequency rises and delivery velocity collapses.

These are not tooling problems.

They are topology problems.

The framework introduced in the book Team Topologies by Matthew Skelton and Manuel Pais provides one of the clearest operational models for designing engineering organisations around flow rather than hierarchy.

The core idea is deceptively simple

Optimise team structures for fast, sustainable software delivery.

This article explains how to apply Team Topologies in practice, identify the organisational anti-patterns slowing your DevOps initiatives, and implement structural changes that improve delivery speed without creating organisational chaos.

Why Team Structure Matters in DevOps

DevOps is often described as a tooling movement.

It is not.

It is fundamentally a sociotechnical systems discipline.

Tooling matters. Automation matters. CI/CD matters.

But organisational communication paths ultimately determine delivery speed.

Conway’s Law famously states:

Organisations design systems that mirror their communication structures.

Meaning:

Fragmented teams create fragmented systems
Bottlenecked organisations create bottlenecked architectures
High-friction communication creates high-friction delivery

Team Topologies provides a practical framework for reducing those organisational bottlenecks systematically.

The 4 Team Types

The Team Topologies model defines four fundamental team types.

Each exists to solve a distinct operational problem.

1. Stream-Aligned Teams

These are the primary delivery teams.

A stream-aligned team owns a flow of business value end-to-end.

Examples:

Payments platform
Customer onboarding
Mobile checkout
Recommendation engine

The key principle:

Single team → owns service lifecycle completely

Including:

Development
Deployment
Operations
Monitoring
Incident response

Characteristics of Strong Stream-Aligned Teams

Healthy stream-aligned teams typically:

Deploy independently
Own production support
Minimise external dependencies
Have clear business alignment
Operate autonomously

Example structure

Team: Payments
Ownership:
- Payment API
- Fraud checks
- Transaction database
- Deployment pipelines
- Monitoring dashboards

This dramatically reduces coordination overhead.

Warning Signs

Stream-aligned teams fail when:

Too many systems are owned
Multiple domains are mixed together
External dependencies dominate delivery
Teams lack operational authority

The result is cognitive overload.

2. Enabling Teams

Enabling teams exist to help other teams improve capabilities.

Not to permanently do the work for them.

Examples:

Kubernetes adoption team
SRE coaching team
Security enablement team
Observability specialists

Their role is temporary acceleration.

Not long-term ownership.

Healthy Enabling Team Behaviour

Good enabling teams:

Teach
Coach
Pair
Document
Reduce friction
Transfer knowledge

Bad enabling teams become outsourced implementation departments.

That destroys scalability.

Example: Kubernetes Enablement

Good pattern:

Enabling Team:
- Creates templates
- Runs workshops
- Helps first deployments
- Coaches incident response

Bad pattern

Every Kubernetes deployment requires enabling team intervention forever

That becomes another bottleneck.

3. Complicated Subsystem Teams

Some domains require deep specialist expertise.

Examples:

ML inference systems
Real-time video encoding
Cryptography engines
High-frequency trading systems

These are cognitively dense domains unsuitable for broad ownership.

Dedicated specialist teams reduce complexity exposure for the rest of the organisation.

Why This Team Type Exists

Without complicated subsystem teams

Every stream-aligned team
↓
Must understand advanced specialist systems

This overwhelms cognitive capacity rapidly.

Example

A recommendation-engine ML platform might require:

Tensor optimisation
GPU scheduling
Feature stores
Embedding pipelines

That expertise does not belong inside every product team.

4. Platform Teams

Platform teams build internal developer platforms.

Their mission

Reduce cognitive load for stream-aligned teams.

Platform teams should operate like product teams.

Not internal ticket queues.

Platform Team Responsibilities

Typical responsibilities:

CI/CD systems
Kubernetes platforms
Observability tooling
Secrets management
Golden deployment paths
Infrastructure templates

Platform-as-a-Product

This concept is critical.

A healthy platform team provides

Self-service capabilities

Not manual intervention.

Good platform

Developer clicks button → environment created

Bad platform

Developer opens Jira ticket → waits 2 weeks

The 3 Interaction Modes

The framework also defines three interaction patterns between teams.

These interaction modes are enormously important operationally.

1. Collaboration Mode

Temporary close cooperation between teams.

Used for:

New capability adoption
Complex integrations
Discovery work

Example

Payments Team ↔ Platform Team

Working together to implement service mesh adoption.

The Key Word: Temporary

Permanent collaboration indicates unclear boundaries.

Collaboration mode should end eventually.

Otherwise dependency chains become permanent.

2. X-as-a-Service Mode

One team provides services consumed independently by others.

This is the desired long-term state for platform teams.

Example

Platform Team → Kubernetes Platform

Consumed self-service by product teams.

Minimal synchronous interaction required.

Signs Your Platform Interface Is Healthy

Healthy X-as-a-Service characteristics:

Well documented
Self-service
Stable APIs
Clear support boundaries
Minimal tickets required

3. Facilitating Mode

Used by enabling teams.

Purpose

Teach capability
Not own capability

Examples:

Security workshops
Incident response coaching
Terraform migration guidance

Facilitating mode transfers knowledge intentionally.

Assessing Your Current Topology: The 6 Key Questions

Most organisations already feel their topology pain intuitively.

This framework helps diagnose it systematically.

Question 1: How Many Teams Are Required for a Deployment?

If the answer exceeds three consistently

Flow efficiency is already degraded.

Question 2: Are Platform Teams Productive or Ticket-Driven?

Platform teams buried in support queues are usually under-designed.

Question 3: Is Production Ownership Clear?

During incidents

Who owns this?

Should never require debate.

Question 4: How Much Cognitive Load Exists Per Team?

Too many technologies, domains, or dependencies create delivery paralysis.

Question 5: How Often Are Teams Waiting on Other Teams?

Dependency-heavy organisations slow exponentially as headcount grows.

Question 6: Are Teams Optimised Around Technology or Business Flow?

Technology-aligned teams often create excessive handoffs.

Business-stream alignment improves delivery velocity dramatically.

Cognitive Load Assessment Framework

Example survey structure

COGNITIVE_LOAD_SURVEY = {
    "domain_complexity": {
        "question": "How well does the team understand the business domain?",
        "red_flag": "< 3"
    },

    "technology_breadth": {
        "question": "How many distinct technologies are maintained?",
        "red_flag": "> 5"
    },

    "dependency_count": {
        "question": "How many teams are required per sprint?",
        "red_flag": "> 3"
    }
}

This kind of lightweight operational telemetry is surprisingly valuable.

The Most Common Team Topologies Anti-Patterns

Most engineering organisations fail in recognisable ways.

The same patterns appear repeatedly.

Anti-Pattern 1: The Shared Services Team Bottleneck

Classic example

Shared DevOps Team

Responsible for:

CI/CD
Kubernetes
Terraform
Monitoring
Networking
Security
Deployments

For every product team.

Result

Centralised dependency bottleneck

Symptoms:

Long ticket queues
Slow onboarding
Deployment delays
Platform burnout

The Real Cost

Shared services teams often become

Organisational rate limiters

Every engineering initiative slows behind them.

Better Model

Replace shared services with:

Stream-aligned ownership
Self-service platforms
Enabling teams
Platform-as-product

Anti-Pattern 2: Platform Teams Without a Defined Interface

Many platform teams say

"We provide Kubernetes."

But what does that actually mean operationally?

Healthy platforms define:

APIs
Golden paths
Support models
Service expectations
Onboarding flows

Without interfaces

Platform becomes tribal knowledge.

Anti-Pattern 3: Enabling Teams That Never Stop Enabling

Enabling teams should create independence.

Not permanent dependency.

Danger signs:

Teams require constant coaching forever
Knowledge transfer never completes
Enablement becomes embedded implementation

At that point the enabling team has failed structurally.

Anti-Pattern 4: Cognitive Load Mismatches

This is one of the most damaging failure modes.

Teams own too much simultaneously:

Multiple languages
Multiple databases
Infrastructure
Security
CI/CD
ML systems
Distributed systems complexity

Eventually

Incident frequency rises
Delivery speed drops
Burnout accelerates

Measuring Cognitive Load

Indicators include

Signal	Warning Threshold
Technologies maintained	> 5
Teams depended on	> 3
Incident ambiguity	Frequent
Deployment complexity	High
Documentation quality	Poor

Cognitive overload is usually visible before collapse occurs.

Planning a Topology Change

Topology redesign is organisational surgery.

Done poorly, it creates chaos.

Done carefully, it dramatically improves flow.

Step 1: Identify Friction Points

Start with:

Deployment delays
Dependency bottlenecks
Ticket queues
Incident ownership confusion
Platform dissatisfaction

Map flow disruptions explicitly.

Step 2: Reduce Team Dependencies

Optimise for

Independent delivery capability

Dependency reduction is usually the highest-ROI organisational improvement.

Step 3: Define Platform Interfaces

Every platform capability should answer:

Who uses this?
How is it consumed?
Is it self-service?
What are support expectations?

Step 4: Transition Gradually

Never reorganise everything simultaneously.

Recommended approach

Pilot topology
↓
Measure outcomes
↓
Expand incrementally

Organisational stability matters.

Measuring the Impact

Topology changes should produce measurable improvements.

Delivery Metrics

Track:

Metric	Why It Matters
Deployment frequency	Measures flow
Lead time	Measures delivery friction
MTTR	Measures operational clarity
Change failure rate	Measures stability

These align closely with DORA metrics.

Cognitive Load Surveys

Run quarterly.

Example

if red_flags >= 3:
    print("Urgent restructuring required")

Even lightweight surveys reveal structural problems surprisingly well.

Platform Satisfaction Scores

Ask stream-aligned teams

How frictionless is the platform?

This single question often exposes platform dysfunction rapidly.

Example Topology Transformation

Before

Developers
↓
Shared DevOps Team
↓
Infrastructure Team
↓
Security Team

Heavy coordination overhead.

Slow deployments.

Unclear ownership.

After

Stream-Aligned Teams
        ↓
Self-Service Platform
        ↓
Enabling Teams

Much faster flow.

Reduced dependencies.

Improved operational autonomy.

Common Mistakes During Team Topologies Adoption

Mistake 1: Renaming Teams Without Changing Responsibilities

Changing titles changes nothing operationally.

Mistake 2: Treating Platform Teams as Infrastructure Operations

Platform teams should optimise developer experience.

Not merely manage Kubernetes clusters.

Mistake 3: Ignoring Cognitive Load

More ownership is not always better.

Mistake 4: Measuring Utilisation Instead of Flow

Highly utilised teams often create slower organisations overall.

Flow efficiency matters more.

Recommended Organisational Architecture

Healthy modern engineering organisations increasingly resemble

Stream-Aligned Teams
        ↓
Platform-as-a-Service
        ↓
Enabling Teams
        ↓
Specialist Subsystem Teams

This structure scales operationally far better than traditional siloed models.

Team Topologies matters because software delivery problems are rarely just technical.

They are organisational.

The framework gives engineering leaders a practical vocabulary for understanding why certain DevOps transformations stall despite heavy investment in tooling and automation.

The most successful organisations consistently optimise for.

Fast flow
Low cognitive load
Clear ownership
Self-service platforms
Minimal dependencies

And those outcomes emerge not from organisational theory alone, but from deliberate topology design.

Because ultimately:

The architecture of your systems
reflects the architecture of your teams.

Always

Secrets Management in Modern DevOps: Vault, IRSA, External Secrets When to Use Each

varun varde — Fri, 15 May 2026 06:11:00 +0000

Secrets management failures rarely begin with malicious intent.

They begin with expediency.

An engineer hardcodes an API key “temporarily.” A .env file gets committed accidentally. A production database password gets shared in Slack during an outage because “we’ll rotate it later.” Eventually those shortcuts accumulate into a sprawling credential catastrophe hidden beneath otherwise competent infrastructure.

The uncomfortable truth is that poor secrets hygiene exists everywhere:

Startups
Scaleups
Enterprises
Banks
Government systems
Fortune 500 infrastructure

The issue is rarely ignorance. It is architectural ambiguity.

Modern DevOps teams now face multiple competing approaches:

Cloud-native identity systems
Kubernetes secret abstractions
Vault
External Secrets Operator
Sealed Secrets
Workload identity federation
Dynamic credentials

Choosing incorrectly creates operational fragility. Choosing well dramatically improves both security and developer experience.

This guide explains when to use each model, where each one fails, and how to evolve from common anti-patterns toward a production-grade secrets architecture without detonating existing workloads.

The Secrets Management Anti-Patterns (and Their Blast Radius)

Before discussing solutions, understand the failure modes.

Because nearly every modern secrets architecture exists to solve one of these disasters.

Anti-Pattern 1: Hardcoded Secrets in Source Code

Example

API_KEY = "sk-prod-293847239847"

This is not merely bad practice.

It is operationally radioactive.

Once committed:

Git history preserves it
Forks replicate it
CI logs may expose it
Developers clone it locally
Backups persist it indefinitely

Even if deleted later.

Anti-Pattern 2: Shared Credentials

Example

prod-admin / password123

Used by:

Developers
CI systems
Automation tools
Contractors

Result

No attribution
No least privilege
No revocation granularity

Shared credentials eliminate accountability entirely.

Anti-Pattern 3: Long-Lived Cloud Access Keys

Example

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

Stored inside:

Jenkins
GitHub Actions
Kubernetes Secrets
Terraform variables

Static credentials eventually leak.

The question is timing, not probability.

Anti-Pattern 4: Kubernetes Secrets Misunderstood as Encryption

Base64 encoding is not encryption.

This surprises people alarmingly often.

Example

echo "cGFzc3dvcmQ=" | base64 -d

Outputs

password

Kubernetes Secrets require additional controls:

Encryption at rest
RBAC
Admission policies
Audit logging

Otherwise they become plaintext credential storage with better branding.

Understanding the Modern Secrets Management Stack

Modern secrets management generally falls into four categories

Each solves different problems.

IRSA / Workload Identity: Cloud-Native Secretless Authentication

This is the most important architectural shift in modern cloud security

Stop distributing credentials.
Start distributing identity.

Instead of giving workloads access keys

Pod → authenticated identity → temporary credentials

No static secrets required.

AWS IRSA (IAM Roles for Service Accounts)

Pods authenticate using Kubernetes service accounts mapped to IAM roles.

Terraform IRSA Role

resource "aws_iam_role" "payment_service" {
  name = "payment-service-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"

    Statement = [{
      Effect = "Allow"

      Principal = {
        Federated = aws_iam_openid_connect_provider.eks.arn
      }

      Action = "sts:AssumeRoleWithWebIdentity"

      Condition = {
        StringEquals = {
          "${replace(
            aws_eks_cluster.main.identity[0].oidc[0].issuer,
            "https://",
            ""
          )}:sub" =
          "system:serviceaccount:payments:payment-service"
        }
      }
    }]
  })
}

Kubernetes Service Account

apiVersion: v1
kind: ServiceAccount

metadata:
  name: payment-service
  namespace: payments

  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/payment-service-role

Pods automatically receive temporary credentials.

No secrets required.

Why IRSA Is Excellent

Advantages:

No static AWS keys
Automatic credential rotation
IAM-native permissions
Short-lived credentials
Excellent auditability

This should be the default model for AWS-native workloads.

GCP Workload Identity Equivalent

GCP uses

Kubernetes Service Account
↔
Google Service Account

Equivalent concept. Different implementation.

Azure Workload Identity

Azure now supports federated workload identity similarly.

The industry is converging on identity federation rather than credential distribution.

This is good.

When IRSA / Workload Identity Is NOT Enough

Cloud-native identity works beautifully for cloud APIs.

It becomes weaker when dealing with:

Databases
Third-party APIs
Cross-cloud systems
Legacy applications
Dynamic credential issuance
Multi-cluster secret orchestration

This is where Vault becomes valuable.

HashiCorp Vault: When You Need More Than Cloud-Native

Vault solves problems identity federation alone cannot.

Especially dynamic secrets.

The Core Vault Capability

Vault does not merely store secrets.

It generates them dynamically.

Example

Application requests PostgreSQL credentials
↓
Vault creates short-lived DB user
↓
Credentials expire automatically

Massive security improvement.

Vault Kubernetes Authentication

Example

vault auth enable kubernetes

Vault Role Example

vault write auth/kubernetes/role/payment-api \
  bound_service_account_names=payment-service \
  bound_service_account_namespaces=payments \
  policies=payment-read \
  ttl=1h

Pods authenticate automatically via Kubernetes identity.

Dynamic Database Credentials

Example

vault read database/creds/payment-role

Returns

{
  "username": "v-token-abc123",
  "password": "generated-secret",
  "lease_duration": 3600
}

Credentials expire automatically after one hour.

When Vault Is the Right Choice

Use Vault when you need

Requirement	Vault
Dynamic secrets	Excellent
Multi-cloud support	Excellent
Fine-grained audit logs	Excellent
PKI management	Excellent
Database credential rotation	Excellent
Secret leasing	Excellent

Vault Tradeoffs

Vault is operationally heavier.

You now manage:

HA clustering
Storage backend
Unseal process
Disaster recovery
Performance replication

Vault is powerful because it solves hard problems.

Hard problems come with operational complexity.

External Secrets Operator: The Kubernetes-Native Abstraction Layer

External Secrets Operator (ESO) is one of the cleanest Kubernetes-native abstractions available today.

Instead of storing secrets directly in Kubernetes

Kubernetes Secret
← synced from →
Vault / AWS Secrets Manager / GCP Secret Manager

Installing ESO

helm install external-secrets external-secrets/external-secrets

AWS Secrets Manager Example

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret

metadata:
  name: payment-api-secret

spec:
  refreshInterval: 1h

  secretStoreRef:
    name: aws-secret-store
    kind: SecretStore

  target:
    name: payment-api-secret

  data:
  - secretKey: api-key
    remoteRef:
      key: prod/payment-api
      property: api_key

Why ESO Is Excellent

Advantages:

Kubernetes-native
GitOps-friendly
Central secret backend
Automatic refresh
Cleaner operational model

ESO is often the best abstraction for Kubernetes workloads.

When ESO Is NOT Enough

ESO synchronises secrets.

It does not generate dynamic credentials.

If you need:

Dynamic DB users
Certificate issuance
Secret leasing
PKI workflows

You still need Vault or equivalent systems.

Sealed Secrets: Simple Offline Encryption for GitOps

Sealed Secrets solve a specific problem elegantly

How do you store encrypted secrets safely in Git?

Sealed Secret Workflow

Developer creates

kubectl create secret generic app-secret

Encrypts

kubeseal --format yaml

Result

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret

Only the cluster controller can decrypt it.

Why Teams Love Sealed Secrets

Benefits:

Simple
GitOps-compatible
Easy onboarding
No external dependency

Where Sealed Secrets Fall Short

Limitations:

Static secrets only
No automatic rotation
Kubernetes-scoped
No dynamic credential issuance

Excellent for smaller GitOps environments.

Less ideal for enterprise-scale secret orchestration.

Secrets Rotation: The Missing Piece Most Implementations Skip

This is the most neglected part of secrets management.

Teams store secrets securely but never rotate them.

Which defeats half the purpose.

Rotation Targets

Rotate regularly:

Secret Type	Rotation Frequency
API keys	30–90 days
DB credentials	Dynamic preferred
TLS certificates	30–90 days
CI tokens	30 days

Vault Dynamic Rotation

Best model

Generate → use → expire automatically

No manual rotation required.

AWS Secrets Manager Rotation

Example Lambda rotation

RotationRules:
  AutomaticallyAfterDays: 30

Common Rotation Failure Mode

Applications caching credentials indefinitely.

Result

Secret rotated
↓
Application breaks

Applications must reload credentials gracefully.

Audit Logging: Knowing Who Accessed What and When

Secrets access without auditing is operational blindness.

Vault Audit Logging

Enable

vault audit enable file file_path=/var/log/vault_audit.log

Every secret request becomes traceable.

AWS CloudTrail

IRSA requests appear in CloudTrail automatically.

This is one reason identity federation is so operationally attractive.

Critical Audit Questions

You should always answer:

Who accessed this secret?
When?
From which workload?
Was it expected?
Was it anomalous?

Without auditability, incident response becomes guesswork.

Migration Playbook: Moving from Hard-Coded to Vault in 4 Weeks

Most organisations cannot migrate instantly.

They need staged evolution.

Week 1: Discovery

Identify:

.env files
Hardcoded credentials
CI secrets
Kubernetes Secrets
Shared accounts

Week 2: Centralisation

Move secrets into:

Vault
AWS Secrets Manager
GCP Secret Manager

Without changing applications yet.

Week 3: Kubernetes Integration

Deploy:

ESO
Vault Agent Injector
IRSA

Start consuming secrets dynamically.

Week 4: Rotation and Cleanup

Rotate:

Old credentials
Shared passwords
Long-lived tokens

Then delete legacy storage completely.

Not “later.”

Immediately.

Multi-Cloud Secrets: Managing Credentials Across AWS, Azure, and GCP

Multi-cloud secrets management becomes operationally difficult quickly.

Recommended Strategy

Use Case	Recommended Tool
AWS-only	IRSA + Secrets Manager
GCP-only	Workload Identity + Secret Manager
Azure-only	Managed Identity + Key Vault
Multi-cloud	Vault

Vault becomes particularly valuable when standardising identity across clouds.

Recommended Enterprise Architecture

Kubernetes Workload
        ↓
IRSA / Workload Identity
        ↓
Vault / Cloud Secret Manager
        ↓
External Secrets Operator
        ↓
Application Runtime

Layered abstractions create operational flexibility.

Common Secrets Management Mistakes

1. Treating Kubernetes Secrets as Secure by Default

They are not.

2. Never Rotating Credentials

Static secrets become permanent liabilities.

3. Using Shared Accounts

Breaks attribution entirely.

4. Giving Vault Excessive Permissions

Vault should broker secrets.

Not become root over everything.

5. Ignoring Audit Logs

Visibility matters as much as encryption.

Modern secrets management is no longer about hiding passwords.

It is about distributing trust safely.

The strongest DevOps environments increasingly follow several principles:

Identity over credentials
Temporary over permanent
Dynamic over static
Automated over manual
Auditable over opaque

IRSA and workload identity eliminate entire classes of cloud credential risk.

Vault enables dynamic, short-lived infrastructure authentication.

External Secrets Operator creates elegant Kubernetes-native integration.

Sealed Secrets simplify GitOps encryption.

Each tool has a legitimate role.

The mistake is not choosing the wrong product.

The mistake is assuming one tool solves every secrets problem equally well.

DevSecOps Pipeline in a Day: Automated Security from Commit to Deploy

varun varde — Tue, 12 May 2026 05:23:00 +0000

Security that happens after deployment is already too late.

By the time a quarterly penetration test discovers hardcoded secrets, vulnerable containers, or publicly exposed infrastructure, the vulnerable code has usually been in production for months. Sometimes years. The remediation backlog grows. Developers lose context. Security becomes bureaucratic archaeology rather than operational engineering.

DevSecOps changes the timing.

Instead of treating security as a gate at the end of delivery, it embeds security checks throughout the software lifecycle.

Commit → Build → Test → Scan → Deploy → Monitor

Every stage becomes an opportunity to reduce risk automatically.

This tutorial builds a complete open-source DevSecOps pipeline in a single day:

Secret detection before commits
SAST on every pull request
Dependency vulnerability scanning
Container image scanning
Terraform and Kubernetes IaC scanning
DAST against staging environments
Centralised vulnerability reporting
Security SLA policies

No enterprise security platform required.

The DevSecOps Security Layer Model Where Each Check Lives

Security works best when distributed.

Not centralised.

Each security control belongs at the earliest operational layer where it can execute effectively.

The Six-Layer Model

Layer 1 → Developer workstation
Layer 2 → Pull request pipeline
Layer 3 → Dependency validation
Layer 4 → Container security
Layer 5 → Infrastructure-as-Code validation
Layer 6 → Runtime application testing

Every layer catches different failure modes.

Why Layering Matters

No single scanner catches everything.

Example

Security becomes resilient through redundancy.

Layer 1: Pre-Commit Hooks — detect-secrets and git-secrets Setup

The cheapest vulnerability to fix is the one that never enters Git history.

Installing Pre-Commit Framework

pip install pre-commit

detect-secrets Configuration

repos:
- repo: https://github.com/Yelp/detect-secrets
  rev: v1.4.0
  hooks:
  - id: detect-secrets

Install hooks

pre-commit install

git-secrets for AWS Credentials

git secrets --install
git secrets --register-aws

Example Detection

AWS_SECRET_ACCESS_KEY detected
Commit rejected

This prevents catastrophic credential leakage before CI even starts.

Why Pre-Commit Security Matters

Secrets committed once often persist forever in Git history.

Even after deletion.

Prevention beats remediation.

Always.

Layer 2: SAST in CI — Semgrep for Application Code

Static Application Security Testing identifies insecure coding patterns before deployment.

Semgrep is exceptionally effective because it balances signal quality with developer usability.

GitHub Actions SAST Workflow

sast:
  runs-on: ubuntu-latest
  steps:
  - uses: actions/checkout@v4

  - name: Semgrep SAST
    uses: returntocorp/semgrep-action@v1
    with:
      config: "p/owasp-top-ten p/python p/javascript"

Example Vulnerability Detection

query = f"SELECT * FROM users WHERE id = {user_input}"

Semgrep flags

Possible SQL injection vulnerability

Custom Security Rules

Production environments eventually require organisation-specific rules.

Example

rules:
- id: no-public-s3
  pattern: '"public-read"'
  message: Public S3 ACL forbidden
  severity: ERROR

Why SAST Must Run on Every PR

Security reviews delayed until release branches create vulnerability bottlenecks.

Fast feedback changes behaviour.

Delayed feedback creates resentment.

Layer 3: SCA — OWASP Dependency-Check and Trivy for Dependencies

Modern applications inherit more code than they write.

Dependency vulnerabilities therefore matter enormously.

OWASP Dependency-Check

dependency-check.sh \
  --project app \
  --scan .

Trivy Dependency Scan

trivy fs .

Example Output

Critical vulnerability:
log4j-core 2.14.1
CVE-2021-44228

Dependency Update Automation

Use Renovate or Dependabot

version: 2
updates:
- package-ecosystem: npm
  schedule:
    interval: daily

Automation reduces vulnerability half-life dramatically.

Layer 4: Container Image Scanning — Trivy in Your Docker Build Pipeline

Containers frequently contain:

Vulnerable OS packages
Unpatched libraries
Misconfigurations
Embedded secrets

Scanning them is mandatory.

Build and Scan Workflow

container-scan:
  runs-on: ubuntu-latest

  steps:
  - uses: actions/checkout@v4

  - name: Build image
    run: docker build -t app:${{ github.sha }} .

  - name: Trivy vulnerability scan
    uses: aquasecurity/trivy-action@master
    with:
      image-ref: app:${{ github.sha }}
      exit-code: '1'
      severity: 'CRITICAL,HIGH'

Example Container Findings

openssl package vulnerable
Severity: HIGH

Distroless Images Reduce Attack Surface

Instead of

FROM ubuntu:22.04

Use

FROM gcr.io/distroless/static

Smaller images. Fewer packages. Fewer CVEs.

Layer 5: IaC Security Scanning — Checkov on Every Terraform Plan

Infrastructure misconfigurations cause some of the most damaging cloud breaches.

IaC scanning catches them before deployment.

Checkov GitHub Action

iac-scan:
  runs-on: ubuntu-latest

  steps:
  - uses: actions/checkout@v4

  - name: Checkov IaC scan
    uses: bridgecrewio/checkov-action@master
    with:
      directory: terraform/
      framework: terraform

Example Terraform Misconfiguration

resource "aws_security_group" "bad" {
  ingress {
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Checkov flags

Security group allows unrestricted ingress

Recommended IaC Policies

Block:

Public S3 buckets
Open security groups
Unencrypted databases
Unencrypted EBS volumes
Wildcard IAM policies

Layer 6: DAST — OWASP ZAP Against Your Staging Environment

DAST validates runtime behaviour.

Unlike SAST, it tests deployed applications directly.

OWASP ZAP Docker Scan

docker run -t owasp/zap2docker-stable \
  zap-baseline.py \
  -t https://staging.example.com

CI Integration Example

- name: ZAP Scan
  run: |
    docker run -t owasp/zap2docker-stable \
      zap-baseline.py \
      -t https://staging.example.com

Vulnerabilities DAST Finds Well

XSS
Missing headers
Insecure cookies
Open redirects
Authentication weaknesses

Why DAST Complements SAST

SAST sees code.

DAST sees behaviour.

You need both.

Centralising Findings in Defect Dojo

Without centralisation, findings scatter across tools and become operational noise.

Defect Dojo consolidates:

SAST results
Dependency scans
Container findings
DAST reports Defect Dojo Deployment

helm install defectdojo defectdojo/defectdojo

Importing Scan Results

curl -X POST https://dojo/api/v2/import-scan/

Why Centralisation Matters

Security programmes fail when visibility fragments.

One dashboard changes operational behaviour.

SLA Policies: How to Treat CRITICAL vs HIGH vs MEDIUM Findings

Not all vulnerabilities deserve identical urgency.

Recommended SLA Model

CI Enforcement Strategy

CRITICAL → Block merge
HIGH → Fail release
MEDIUM → Warn only

Security governance must remain operationally realistic.

Overly aggressive policies create bypass behaviour.

Measuring DevSecOps Effectiveness Mean Time to Remediation

Security programmes require measurable outcomes.

Core Metrics
Mean Time to Remediation (MTTR)

Discovery → Remediation

Shorter is better.

Vulnerability Escape Rate

How many vulnerabilities reach production?

False Positive Rate

If scanners create excessive noise

Developers stop trusting alerts

Signal quality matters enormously.

Coverage Metrics

Track:

Repositories scanned
Terraform coverage
Container coverage
Dependency scan adoption

Full GitHub Actions DevSecOps Workflow

name: DevSecOps Pipeline

on: [push, pull_request]

jobs:

  secrets-scan:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: TruffleHog
      uses: trufflesecurity/trufflehog@main

  sast:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Semgrep
      uses: returntocorp/semgrep-action@v1

  dependency-scan:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Trivy FS Scan
      run: trivy fs .

  container-scan:
    runs-on: ubuntu-latest
    steps:
    - run: docker build -t app:${{ github.sha }} .

    - name: Trivy Image Scan
      run: trivy image app:${{ github.sha }}

  iac-scan:
    runs-on: ubuntu-latest
    steps:
    - name: Checkov
      uses: bridgecrewio/checkov-action@master

  dast:
    runs-on: ubuntu-latest
    steps:
    - name: OWASP ZAP
      run: |
        docker run -t owasp/zap2docker-stable \
        zap-baseline.py \
        -t https://staging.example.com

Common DevSecOps Mistakes

1. Blocking Everything Immediately

Teams bypass pipelines if friction becomes unbearable.

Adopt incrementally.

2. Ignoring False Positives

Poor signal quality destroys developer trust.

3. Treating Security as Separate from Engineering

Security tooling must integrate into existing workflows.

Not create parallel ones.

4. No Ownership Model

Findings without owners become backlog sediment.

DevSecOps is not about inserting security gates into delivery pipelines.

It is about making security part of normal engineering behaviour.

The most successful DevSecOps environments share several characteristics:

Fast feedback
Automated enforcement
Low-friction tooling
Developer-visible results
Incremental adoption

Security stops being ceremonial compliance theatre and becomes operational engineering.

And that is the critical shift.

Because modern software delivery moves too quickly for security reviews performed weeks after deployment.

The only scalable model is continuous security at continuous delivery speed.

FinOps for DevOps Engineers: The Complete Cloud Cost Optimisation Playbook

varun varde — Fri, 08 May 2026 12:03:16 +0000

Cloud bills rarely explode because of one catastrophic decision. They grow incrementally. Quietly. A forgotten load balancer here. Overprovisioned Kubernetes nodes there. NAT Gateway traffic multiplying invisibly in the background like fiscal mold behind drywall.

Most organisations approach FinOps as a finance exercise. That is a strategic mistake.

The engineers provisioning infrastructure are the same engineers best positioned to optimise it. DevOps teams control autoscaling, storage policies, networking topology, observability retention, and workload scheduling. They are not adjacent to cloud cost optimisation. They are the operational epicentre of it.

This playbook focuses on practical FinOps implementation for DevOps and platform engineers. Not abstract governance theory. Actual engineering patterns that reduce spend without degrading reliability.

The optimisation path is organised by return on investment. Start with visibility. Then tackle compute, storage, networking, Kubernetes, and finally governance automation.

Part 1: Visibility First — Tagging Standards and Cost Attribution

You cannot optimise what you cannot attribute.

Most cloud environments fail at cost management because nobody knows which team owns what.

The Minimum Viable Tagging Standard

Every resource should contain

tags = {
  team         = "platform-engineering"
  environment  = "production"
  application  = "checkout-api"
  cost-centre  = "ENG-042"
  owner        = "payments-team"
}

Why Tags Matter

Without tags

Cloud bill = giant undifferentiated blob

With tags

Cloud bill = attributable operational data

This changes engineering behaviour immediately.

AWS Cost Allocation Tags

Enable them explicitly

aws ce list-cost-allocation-tags

Then activate

Billing Console → Cost Allocation Tags → Activate

Cost Dashboard Strategy

Build dashboards around:

Cost by team
Cost by environment
Cost by service
Week-over-week growth
Top anomalous resources

Part 2: Compute Optimisation — Rightsizing, Spot, Graviton

Compute is usually the largest controllable expense category.

And most environments are dramatically oversized.

Rightsizing EC2 Instances

Example

m5.4xlarge
Average CPU: 9%

This is not infrastructure. It is financial leakage.

Identify Idle Instances

Using CloudWatch

Spot Instances

Spot pricing can reduce costs by 70–90%.

Perfect for:

CI runners
Batch jobs
Non-critical workloads
Kubernetes worker nodes

Terraform Spot Example

resource "aws_instance" "spot_worker" {
  instance_type = "m7g.large"

  instance_market_options {
    market_type = "spot"
  }
}

AWS Graviton Migration

Graviton instances routinely reduce compute costs by 20–40%.

Migration Candidate Checklist

Best workloads:

Stateless APIs
Containers
Node.js
Go
Java 17+

Kubernetes Node Group Example

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

managedNodeGroups:
  - name: graviton-workers
    instanceType: m7g.large

Part 3: Storage Optimisation — S3 Tiers, EBS, Lifecycle Policies

Storage inefficiency compounds silently over years.

S3 Lifecycle Policies

The fastest storage win in AWS.

Terraform Lifecycle Policy

resource "aws_s3_bucket_lifecycle_configuration" "cost_optimised" {
  bucket = aws_s3_bucket.data.id

  rule {
    id     = "archive-old-data"
    status = "Enabled"

    transition {
      days          = 30
      storage_class = "INTELLIGENT_TIERING"
    }

    transition {
      days          = 90
      storage_class = "GLACIER_IR"
    }

    transition {
      days          = 365
      storage_class = "DEEP_ARCHIVE"
    }
  }
}

EBS Optimisation

Common waste patterns:

Detached volumes
Oversized gp3 disks
Unused snapshots

Find Unattached Volumes

aws ec2 describe-volumes \
  --filters Name=status,Values=available

Unless compliance requires otherwise.

Part 4: Networking Cost Reduction — NAT Gateway, VPC Endpoints, Data Transfer

Networking costs surprise almost everyone.

Especially NAT Gateways.

NAT Gateway Optimisation

NAT Gateway charges include:

Hourly fee
Per-GB transfer fee

Large clusters can spend thousands monthly on NAT traffic alone.

Replace NAT Traffic with VPC Endpoints

Example

resource "aws_vpc_endpoint" "s3" {
  vpc_id       = aws_vpc.main.id
  service_name = "com.amazonaws.us-east-1.s3"
}

This eliminates NAT transfer charges for S3 traffic.

Reduce Cross-AZ Traffic

Hidden cost source:

Service A → AZ-1
Service B → AZ-2

Every request incurs transfer cost.

Kubernetes Affinity Rules

topologySpreadConstraints:
- topologyKey: topology.kubernetes.io/zone

Keep chatty services co-located.

Part 5: Database Cost Optimisation — RDS Rightsizing, Aurora Serverless, Read Replica Pruning

Databases are expensive because teams fear touching them.

Reasonably so.

RDS Rightsizing

Monitor:

CPU
Connections
IOPS
Memory pressure

Example Downsize

db.r6g.4xlarge → db.r6g.xlarge

Often invisible to applications.

Massively visible to finance.

Aurora Serverless v2

Ideal for:

Variable workloads
Internal APIs
Intermittent services

Terraform Example

serverlessv2_scaling_configuration {
  min_capacity = 0.5
  max_capacity = 8
}

Read Replica Cleanup

Common anti-pattern

Temporary read replica
→ never removed
→ costs persist forever

Audit quarterly.

Part 6: Reserved Instances & Savings Plans — When to Buy and How Much

Savings Plans are powerful when used correctly.

Dangerous when guessed incorrectly.

Recommended Strategy

Start conservative.

Target:

60–70% baseline utilisation coverage

Never 100%.

Compute Savings Plans

Best default option.

Flexible across:

Instance families
Regions
Compute types

AWS Recommendation API

aws ce get-savings-plans-purchase-recommendation

Use actual usage history.

Not optimism.

Part 7: Kubernetes Cost Optimisation — Bin Packing, Cluster Autoscaler, Spot Node Groups

Kubernetes amplifies both efficiency and waste.

Bin Packing

Underutilised nodes are financial dead weight.

Resource Requests Matter

Bad:

requests:
  cpu: "4"

Actual usage:

300m

Goldilocks Recommendation Tool

kubectl install goldilocks

Automatically suggests request sizing.

Cluster Autoscaler

--balance-similar-node-groups=true

Removes idle nodes dynamically.

Spot Node Groups

Example

capacityType: SPOT

Excellent for:

Stateless apps
Batch workers
CI runners

Part 8: Monitoring Cost Creep — Alerting on Unexpected Spend Increases

Cost optimisation without monitoring regresses rapidly.

Budget Alerts

AWS Example

aws budgets create-budget

Prometheus Cost Alert

groups:
- name: cloud_cost_alerts
  rules:
  - alert: MonthlySpendSpike
    expr: increase(cloud_cost_total[24h]) > 1000

Slack Notification Example

import requests

requests.post(
  webhook_url,
  json={"text": "Cloud spend increased unexpectedly"}
)

Immediate visibility changes behaviour.

Part 9: The Monthly Cost Review Checklist

The best FinOps teams operationalise review cadence.

Monthly Checklist
Compute

Idle instances removed
Rightsizing opportunities reviewed
Spot coverage audited

Storage

Snapshot retention reviewed
Glacier transitions verified
Orphaned volumes deleted

Kubernetes

Node utilisation checked
Resource requests audited
Cluster Autoscaler effectiveness reviewed

Networking

NAT Gateway spend analysed
Cross-region traffic reviewed

Databases

Read replicas validated
Aurora scaling reviewed

Governance

Untagged resources identified
Budget alerts tested

Appendix: Azure and GCP Equivalents

Compute

FinOps is not about making infrastructure cheap.

It is about making infrastructure intentional.

The most effective DevOps teams treat cloud cost as an engineering metric alongside latency, reliability, and deployment frequency.

Because every oversized node, forgotten snapshot, or unnecessary NAT transfer represents engineering inefficiency expressed financially.

The progression usually looks like this:

Visibility → Attribution → Accountability → Optimisation

Without visibility, optimisation is guesswork.

Without attribution, accountability disappears.

Without accountability, cloud spend becomes entropy.

But when engineers own both infrastructure reliability and infrastructure economics, something powerful happens:

Systems become leaner.
Architectures become cleaner.
And cloud bills stop being monthly surprises.

The FinOps Starter Kit: Making Cloud Cost Visible in 5 Days

varun varde — Fri, 01 May 2026 08:29:13 +0000

Most cloud cost advice starts at the wrong layer. It jumps straight into optimization tactics Reserved Instances, Spot capacity, aggressive rightsizing without first addressing the more fundamental problem: visibility.

Because without visibility, optimization becomes guesswork. And guesswork is expensive.

This guide takes a different approach. Five days. No third-party FinOps platforms. Just native tooling, deliberate structure, and a system engineers will actually use.

Day 1: Tagging Strategy The Foundation Everything Else Depends On

Every meaningful cost analysis begins with attribution. Without tags, cost data is a monolith. With tags, it becomes dimensional.

Core Tagging Model

A minimal, effective tagging schema:

{
  "team": "platform",
  "service": "auth-api",
  "environment": "production",
  "owner": "team-lead"
}

Enforcing Tags at Resource Creation

aws ec2 run-instances \
  --image-id ami-123456 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=team,Value=platform},{Key=service,Value=auth-api}]'

Tag Compliance Check

aws resourcegroupstaggingapi get-resources \
  --tag-filters Key=team

Why This Matters

Tags are not metadata. They are the index keys for your cost database.

No tags → no attribution → no accountability.

Day 2: AWS Cost Explorer API — Pulling Cost Data Programmatically

The console is fine for humans. Systems need APIs.

Basic Cost Query

import boto3

ce = boto3.client('ce')

response = ce.get_cost_and_usage(
    TimePeriod={"Start": "2026-04-01", "End": "2026-04-30"},
    Granularity="DAILY",
    Metrics=["UnblendedCost"]
)

Group by Service and Team Tag

response = ce.get_cost_and_usage(
    TimePeriod={"Start": "2026-04-01", "End": "2026-04-30"},
    Granularity="DAILY",
    Metrics=["UnblendedCost"],
    GroupBy=[
        {"Type": "DIMENSION", "Key": "SERVICE"},
        {"Type": "TAG", "Key": "team"}
    ]
)

Key Insight

Cost data is delayed (~24h), but still actionable.

This becomes your source of truth. Everything else builds on it.

Day 3: Building Per-Service Cost Dashboards in Grafana

Raw data is inert. Visualization activates it.

Architecture

AWS Cost Explorer → Export Script → JSON/Prometheus → Grafana

Example Export Script

import json

data = response["ResultsByTime"]

with open("cost.json", "w") as f:
    json.dump(data, f)

Prometheus Metric Format

aws_cost{service="EC2",team="platform"} 123.45

Grafana Panel Query

sum by(service) (aws_cost)

Dashboard Views

Cost per service
Cost per team
Daily trend lines
Top 10 spenders

Good dashboards don’t overwhelm. They illuminate.

Day 4: Anomaly Detection Alerting When Cost Spikes Unexpectedly

Spikes happen. Some are valid. Others are not.

Detection must be immediate.

Simple Threshold Alert

aws_cost_daily > 500

Deviation-Based Alert

aws_cost_daily > avg_over_time(aws_cost_daily[7d]) * 1.5

CloudWatch Anomaly Detection

aws cloudwatch put-anomaly-detector \
  --metric-name EstimatedCharges \
  --namespace AWS/Billing

Alert Routing

Alert → SNS → Slack / Email

Short spikes matter. Long drifts matter more.

Both need visibility.

Day 5: The Weekly Cost Digest Automated Slack Report Per Team

Dashboards are passive. Digests are proactive.

Engineers rarely check dashboards. They read Slack.

Weekly Cost Digest Script

# cost_digest.py Weekly per-team cost report to Slack
import boto3, json, datetime
from slack_sdk import WebClient

ce = boto3.client('ce', region_name='us-east-1')
slack = WebClient(token="YOUR_SLACK_TOKEN")

def get_team_costs(team_tag: str, days: int = 7) -> dict:
    end = datetime.date.today()
    start = end - datetime.timedelta(days=days)

    resp = ce.get_cost_and_usage(
        TimePeriod={"Start": str(start), "End": str(end)},
        Granularity="DAILY",
        Filter={"Tags": {"Key": "team", "Values": [team_tag], "MatchOptions": ["EQUALS"]}},
        Metrics=["UnblendedCost"],
        GroupBy=[{"Type": "DIMENSION", "Key": "SERVICE"}]
    )

    totals = {}
    for result in resp["ResultsByTime"]:
        for group in result["Groups"]:
            svc = group["Keys"][0]
            cost = float(group["Metrics"]["UnblendedCost"]["Amount"])
            totals[svc] = totals.get(svc, 0) + cost

    return totals

def post_digest(team: str, channel: str):
    costs = get_team_costs(team)
    total = sum(costs.values())

    lines = [
        f"*Weekly Cloud Cost Digest — Team: {team}*",
        f"Total (last 7 days): *${total:,.2f}*",
        ""
    ]

    for svc, cost in sorted(costs.items(), key=lambda x: -x[1])[:8]:
        lines.append(f" • {svc}: ${cost:,.2f}")

    slack.chat_postMessage(channel=channel, text="\n".join(lines))

# Run weekly via EventBridge scheduled rule
post_digest("platform-team", "#platform-costs")

Scheduling with EventBridge

aws events put-rule \
  --schedule-expression "cron(0 9 ? * MON *)"

This creates a ritual. A cadence. Cost becomes visible and social.

Bonus: Cost-per-Request Metrics Using CloudWatch + Lambda

Absolute cost is useful. Unit cost is transformative.

Custom Metric Example

import boto3

cloudwatch = boto3.client('cloudwatch')

cloudwatch.put_metric_data(
    Namespace='AppMetrics',
    MetricData=[
        {
            'MetricName': 'CostPerRequest',
            'Value': 0.002
        }
    ]
)

Formula

Cost per request = Total service cost / Total requests

Now teams optimize efficiency not just spend.

Azure and GCP Equivalents

Azure

Cost Management API
Azure Monitor
Tags via Resource Manager

GCP

Billing Export to BigQuery
Looker Studio dashboards
Labels for resource tagging

The principles remain identical. Only the APIs differ.

Common Tagging Mistakes (and How to Fix Them)

1. Inconsistent Tag Keys

team vs Team vs TEAM

Fix: Enforce via policy.

2. Missing Tags on Critical Resources

Fix: Use SCPs or IAM policies

{
  "Effect": "Deny",
  "Action": "ec2:RunInstances",
  "Condition": {
    "Null": {
      "aws:RequestTag/team": "true"
    }
  }
}

3. Over-Tagging

Too many tags dilute clarity.

Fix: Keep it minimal. Intentional.

FinOps does not begin with optimization. It begins with visibility.

In five days:

Costs become attributable
Dashboards become actionable
Alerts become immediate
Engineers become accountable

And something subtle happens.

Cost stops being a finance concern. It becomes an engineering signal.

That shift quiet, structural, and profound is where real savings begin.

Your First LLMOps Pipeline: From Prompt to Production in One Sprint

varun varde — Tue, 21 Apr 2026 04:37:00 +0000

AI applications don’t behave like traditional systems. They don’t fail cleanly. They don’t produce identical outputs for identical inputs. And they don’t lend themselves to binary testing pass or fail.

Instead, they operate in gradients. Probabilities. Trade-offs.

That is precisely why applying standard DevOps or MLOps practices without adaptation often leads to brittle pipelines and unreliable outcomes.

This guide walks through a complete LLMOps pipeline practical, production-ready, and deployable within a single sprint.

LLMOps vs MLOps vs DevOps - The Operational Model Differences

Traditional DevOps assumes determinism

Input → Code → Output (predictable)

MLOps introduces probabilistic behavior but still focuses on trained models

Input → Model → Prediction (statistical)

LLMOps shifts the paradigm further

Input → Prompt + Model → Generated Output (non-deterministic)

Key distinctions

Outputs vary even with identical inputs
Prompt design is as critical as code
Latency and cost are tied to tokens, not just compute

This necessitates new operational primitives.

Prompt Versioning: Treating Prompts as Code

Prompts are no longer ephemeral strings. They are artifacts.

Store them in Git

/prompts/
  summarization/
    v1.0.0.txt
    v1.1.0.txt

Example prompt

# v2.3.1
Summarize the following text in 3 bullet points with a professional tone:

Reference prompts explicitly in code

PROMPT_VERSION = "v2.3.1"

with open(f"prompts/summarization/{PROMPT_VERSION}.txt") as f:
    prompt_template = f.read()

Never use latest. Ambiguity is the enemy of reproducibility.

Evaluation Frameworks: How to Test LLM Outputs

Testing LLMs requires nuance. Exact matches are rare. Evaluation must be semantic.

Example using a scoring function

def evaluate_output(expected, actual):
    return similarity_score(expected, actual) > 0.85

Dataset-driven testing

[
  {
    "input": "Explain Kubernetes",
    "expected": "Container orchestration platform"
  }
]

Run batch evaluations

python evaluate.py --dataset test_cases.json

Metrics to track

Relevance
Coherence
Hallucination rate

Testing becomes statistical—not absolute.

CI/CD for LLM Applications: What to Run on Every PR

CI pipelines must evolve.

A minimal LLM CI pipeline

name: LLM CI

on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - run: python evaluate.py
      - run: python lint_prompts.py
      - run: python cost_estimator.py

Checks include

Prompt syntax validation
Regression detection in outputs
Cost estimation per request

A failing evaluation blocks the merge. Quality is enforced early.

Deployment Patterns: Blue-Green and Canary

Non-determinism demands cautious rollout.

Blue-Green Deployment

version: v1 (blue)
version: v2 (green)

Switch traffic atomically.

Canary Deployment

traffic:
  v1: 90%
  v2: 10%

Monitor performance before full rollout.

Example Kubernetes snippet

apiVersion: networking.k8s.io/v1
kind: Ingress
spec:
  rules:
    - http:
        paths:
          - backend:
              service:
                name: llm-service-v2

Observe behavior before committing fully.

Observability: Traces, Latency, and Token Costs

Observability must capture more than uptime.

Tracing

from opentelemetry import trace

tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("llm_request"):
    response = call_llm()

Metrics

histogram_quantile(0.95, rate(llm_latency_seconds_bucket[5m]))

Cost Tracking

sum(increase(llm_tokens_total[1h])) * 0.000002

Dashboards should answer

How fast?
How expensive?
How reliable?

Guardrails: Output Validation and Fallback Chains

LLMs can produce unexpected outputs. Guardrails mitigate risk.

Validation Example

def validate_output(output):
    return "forbidden_word" not in output

Fallback Chain

try:
    response = call_primary_model()
except:
    response = call_secondary_model()

Content Filtering

if toxicity_score(output) > 0.7:
    return "Content not allowed"

Guardrails are not optional. They are essential.

Cost Controls: Token Budgets and Rate Limiting

Costs scale with usage. Left unchecked, they escalate rapidly.

Token Limits

MAX_TOKENS = 2000

Rate Limiting

if requests_per_minute > 100:
    reject_request()

Budget Enforcement

if monthly_tokens > budget:
    disable_non_critical_features()

Cost awareness must be embedded in the system—not retrofitted.

Human-in-the-Loop Workflows

For high-stakes decisions, automation alone is insufficient.

Approval Workflow

LLM Output → Human Review → Final Decision

Queue System

if confidence_score < 0.8:
    send_to_review_queue()

Humans provide judgment where models provide probability.

Complete Example: Production-Ready LLM Pipeline on Kubernetes

# llm-pipeline-values.yaml — Kubernetes deployment with cost + observability
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-service
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: llm-service
          image: your-org/llm-service:v1.2.0
          env:
            - name: MAX_TOKENS_PER_REQUEST
              value: "2000"
            - name: MONTHLY_TOKEN_BUDGET
              value: "10000000"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-collector:4317"
            - name: PROMPT_VERSION
              value: "v2.3.1"
          resources:
            requests:
              memory: "256Mi"
              cpu: "100m"
            limits:
              memory: "512Mi"
              cpu: "500m"
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: llm-cost-alerts
spec:
  groups:
    - name: llm_cost
      rules:
        - alert: LLMDailySpendHigh
          expr: sum(increase(llm_tokens_total[24h])) * 0.000002 > 50
          for: 5m
          annotations:
            summary: "LLM daily spend exceeding $50 threshold"

This configuration encapsulates

Versioned prompts
Observability hooks
Cost safeguards
Scalable deployment

LLMOps is not an extension of DevOps. It is a rethinking.

Systems are no longer deterministic. Testing is no longer binary. Costs are no longer predictable.

Yet, with the right structure versioning, evaluation, observability, and control—the uncertainty becomes manageable. Even advantageous.

A well-designed LLMOps pipeline does not eliminate unpredictability. It harnesses it.

Building Production-Grade Observability: OpenTelemetry + Grafana Stack

varun varde — Tue, 14 Apr 2026 09:05:05 +0000

Stop guessing what's broken in production. Here's a complete, deploy-it-this-week observability stack built on OpenTelemetry and Grafana — the same stack I've deployed for three clients in the last 18 months.

This isn't a toy setup. This is production-grade: traces, metrics, and logs unified under a single pane of glass, with auto-instrumentation for the most common runtimes, alerting that pages on symptoms not causes, and dashboards your non-SRE teammates can actually read.

What you'll build:

OpenTelemetry Collector (gateway mode) for vendor-agnostic telemetry collection
Grafana Tempo for distributed tracing
Prometheus + Grafana Mimir for metrics at scale
Loki for structured log aggregation
Grafana dashboards with pre-built SLO panels
AlertManager rules tied to error budgets

Prerequisites: Kubernetes 1.25+, Helm 3, basic familiarity with YAML. Estimated time: 3–5 hours end to end.

Why OpenTelemetry? The vendor-lock argument settled once and for all

You’ve heard it before: “Just use Datadog.” Then the bill arrives. Or “Use Prometheus alone.” Then you lose traces.

OpenTelemetry (OTel) is the single CNCF standard for generating and exporting telemetry data. Here’s why it wins:

One instrumentation, many backends: Instrument your app once with OTel SDKs. Send to Tempo, Jaeger, Datadog, or New Relic simultaneously.
No vendor lock-in: Your telemetry data remains in your control (S3 for traces, block storage for metrics).
Automatic context propagation: Trace IDs flow seamlessly across services, even across different languages (Java → Python → Node.js).
Future-proof: New backends emerge? Point your OTel Collector there. No code changes.

The bottom line: OTel is the USB-C of observability. Stop writing custom exporters.

Architecture overview: Collector, Backends, Visualization

Here’s what you’re deploying:

[Your App] --(OTLP)--> [OTel Collector (Gateway)] --+--> [Tempo] (traces)
                                                      +--> [Mimir] (metrics)
                                                      +--> [Loki] (logs)
                                                              |
                                                         [Grafana] (visualization)
                                                              |
                                                       [AlertManager] (paging)

OTel Collector (Gateway mode): Receives OTLP from all services. Validates, batches, and routes telemetry. Single ingress point.
Tempo: Object-storage-backed tracing. Cheap, scalable, no indexing costs.
Mimir: Horizontally scalable Prometheus-compatible metrics store.
Loki: Log aggregation with low-cost object storage.
Grafana: Unified UI with Explore, dashboards, and alerting.
AlertManager: Deduplicates, groups, and routes alerts to PagerDuty/Slack.

Storage requirements (minimal): 50GB for Loki, 100GB for Tempo (can use S3/GCS/MinIO), 50GB for Mimir.

Installing the OTel Collector (gateway mode Helm values)

Create otel-collector-values.yaml

mode: deployment   # gateway mode (as opposed to daemonset for agent mode)

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  processors:
    batch:
      timeout: 1s
      send_batch_size: 1024
    memory_limiter:
      check_interval: 1s
      limit_mib: 512
    attributes:
      actions:
        - key: environment
          value: production
          action: upsert

  exporters:
    otlp/tempo:
      endpoint: "tempo-distributor:4317"
      tls:
        insecure: true
    prometheusremotewrite/mimir:
      endpoint: "http://mimir-distributor:8080/api/v1/push"
    loki:
      endpoint: "http://loki-gateway:3100/loki/api/v1/push"

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch, attributes]
        exporters: [otlp/tempo]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [prometheusremotewrite/mimir]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [loki]

Deploy

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm upgrade --install otel-collector open-telemetry/opentelemetry-collector -f otel-collector-values.yaml

Auto-instrumentation: Java, Python, Node.js, Go

No code changes for traces/metrics/logs. Use OTel's auto-instrumentation agents.

Java (Spring Boot, any JVM app)

ENV JAVA_TOOL_OPTIONS="-javaagent:/otel/opentelemetry-javaagent.jar"
ENV OTEL_SERVICE_NAME=payment-service
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

Python (Django, Flask, FastAPI)

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
otel-instrument \
  --service_name checkout-service \
  --exporter_otlp_endpoint http://otel-collector:4317 \
  python app.py

Node.js (Express, NestJS)

npm install @opentelemetry/auto-instrumentations-node
npx opentelemetry-instrument \
  --service_name=api-gateway \
  --exporter_otlp_endpoint=http://otel-collector:4317 \
  node server.js

Go (manual instrumentation required, but minimal)

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
)

func initTracer() {
    exporter, _ := otlptracegrpc.New(ctx, 
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure())
    // ... standard setup (5 lines)
}

Verify: Check Collector logs for TraceID spans.

Deploying Tempo for distributed tracing

Tempo is designed for cost-effective tracing. It stores traces in object storage (S3/MinIO) and indexes only by trace ID.

tempo-values.yaml

tempo:
  storage:
    trace:
      backend: s3
      s3:
        bucket: tempo-traces
        endpoint: minio.minio:9000
        access_key: "minioadmin"
        secret_key: "minioadmin"
        insecure: true
      pool:
        max_workers: 100
        queue_depth: 10000

  overrides:
    defaults:
      ingestion:
        rate_limit_bytes: 15000000   # 15MB/s
        burst_size_bytes: 20000000

distributor:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"

Deploy

helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install tempo grafana/tempo -f tempo-values.yaml

Query Tempo from Grafana: Add data source → Tempo → URL: http://tempo-query-frontend:16686

Prometheus + Mimir for long-term metrics storage

Mimir replaces single-instance Prometheus. It provides horizontal scaling, replication, and long-term retention.

mimir-values.yaml

mimir:
  structuredConfig:
    blocks_storage:
      backend: s3
      s3:
        endpoint: minio.minio:9000
        bucket_name: mimir-blocks
        access_key_id: "minioadmin"
        secret_access_key: "minioadmin"
        insecure: true
    ingester:
      ring:
        replication_factor: 3   # for HA
    ruler:
      rule_path: /data/rules
      alertmanager_url: http://alertmanager:9093

  ingester:
    replicas: 3
  distributor:
    replicas: 2
  querier:
    replicas: 2

Deploy

helm upgrade --install mimir grafana/mimir -f mimir-values.yaml

Migrate existing Prometheus data

promtool tsdb create-blocks-from-rules --rules-file=recording-rules.yaml data/

Then point Prometheus remote write to http://mimir-distributor:8080/api/v1/push.

Loki for log aggregation with structured querying

Loki is like Prometheus for logs. It indexes only labels, not full text, making it cheap at scale.

loki-values.yaml

loki:
  storage:
    type: s3
    s3:
      endpoint: minio.minio:9000
      bucketnames: loki-chunks
      access_key_id: "minioadmin"
      secret_access_key: "minioadmin"
      s3forcepathstyle: true
      insecure: true

  schemaConfig:
    configs:
      - from: 2024-01-01
        store: boltdb-shipper
        object_store: s3
        schema: v12
        index:
          prefix: loki_index_
          period: 24h

  limits_config:
    ingestion_rate_mb: 10
    ingestion_burst_size_mb: 20
    max_global_streams_per_user: 10000

  chunk_store_config:
    max_look_back_period: 672h  # 28 days

Deploy

helm upgrade --install loki grafana/loki -f loki-values.yaml

Query example (LogQL)

{namespace="production", app="payment-service"} |= "error" 
| json 
| latency_ms > 500 
| line_format "{{.trace_id}} - {{.message}}"

Grafana: Connecting all three data sources

grafana-values.yaml

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus-Mimir
        type: prometheus
        url: http://mimir-query-frontend:8080/prometheus
        access: proxy
        isDefault: true

      - name: Tempo
        type: tempo
        url: http://tempo-query-frontend:16686
        access: proxy
        jsonData:
          tracesToLogs:
            datasourceUid: 'loki'
            tags: ['service.name', 'pod']
          serviceMap:
            enabled: true

      - name: Loki
        type: loki
        url: http://loki-gateway:3100
        access: proxy
        jsonData:
          derivedFields:
            - name: trace_id
              matcherRegex: 'trace_id=(\w+)'
              url: '$${__value.raw}'
              datasourceUid: 'tempo'

dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: 'slo'
        orgId: 1
        folder: 'SLO Dashboards'
        type: file
        options:
          path: /var/lib/grafana/dashboards

Deploy

helm upgrade --install grafana grafana/grafana -f grafana-values.yaml

Test correlation: In Loki, find a log with trace_id=abc123. Click it → jumps to Tempo trace. In Tempo, see affected service → jumps to Mimir metrics for that service.

Building your first SLO dashboard (template included)

Save as slo-dashboard.json and mount into Grafana

{
  "title": "SLO Dashboard - Payment Service",
  "panels": [
    {
      "title": "Availability (30d SLI)",
      "targets": [{
        "expr": "sum(rate(http_requests_total{status!~'5..'}[$__range])) / sum(rate(http_requests_total[$__range]))",
        "legendFormat": "Availability SLI"
      }],
      "thresholds": [
        {"color": "red", "value": null, "op": "lt", "valueType": "absolute", "value": 0.995},
        {"color": "yellow", "value": null, "op": "lt", "valueType": "absolute", "value": 0.999},
        {"color": "green", "value": null, "op": "gte", "valueType": "absolute", "value": 0.999}
      ]
    },
    {
      "title": "Error Budget Remaining (30d)",
      "targets": [{
        "expr": "(1 - (sum(rate(http_requests_total{status=~'5..'}[30d])) / sum(rate(http_requests_total[30d])))) / 0.999",
        "legendFormat": "Budget remaining"
      }],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "min": 0,
          "max": 1,
          "color": {"mode": "thresholds"},
          "thresholds": [
            {"color": "red", "value": null, "op": "lt", "value": 0.7},
            {"color": "yellow", "value": null, "op": "lt", "value": 0.9},
            {"color": "green", "value": null, "op": "gte", "value": 0.9}
          ]
        }
      }
    },
    {
      "title": "Latency P99 (30d SLI)",
      "targets": [{
        "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[$__range])) by (le))",
        "legendFormat": "P99 latency"
      }]
    }
  ]
}

SLO math explained

Availability target: 99.9% → error budget = 0.1% of requests can fail.
Budget remaining: (actual_availability - target) / (1 - target) → 1.0 means on track, 0 means exhausted.

AlertManager: Alerting on symptoms, not causes

Bad alert: "CPU on pod payment-7d8f9 is 92%" (cause)
Good alert: "Payment service error budget exhausted" (symptom)

alertmanager-config.yaml

route:
  group_by: ['alertname', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'pagerduty-critical'
  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical
      continue: false
    - match:
        severity: warning
      receiver: slack-warnings

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: <your-pd-key>
        severity: critical
  - name: 'slack-warnings'
    slack_configs:
      - api_url: <webhook>
        channel: '#alerts-warning'

Prometheus alerting rule example (slo-alerts.yaml)

groups:
  - name: slo
    rules:
      - alert: ErrorBudgetExhausted
        expr: |
          (1 - (sum(rate(http_requests_total{status=~"5.."}[30d])) 
          / sum(rate(http_requests_total[30d])))) / 0.999 < 0.2
        for: 5m
        labels:
          severity: critical
          service: "{{$labels.service}}"
        annotations:
          summary: "Error budget for {{$labels.service}} is below 20%"
          description: "Remaining budget: {{$value | humanizePercentage}}"

Deploy

kubectl create configmap alertmanager-config --from-file=alertmanager.yaml=alertmanager-config.yaml
helm upgrade --install prometheus prometheus-community/prometheus \
  --set alertmanager.enabled=true \
  --set alertmanager.configFromSecret=alertmanager-config

The 3 dashboards every on-call engineer needs

Stop building 50-panel dashboards. Start with these three.

Dashboard 1: Service Health (RED method)

Rate (requests per second) per endpoint
Errors (5xx rate, grouped by status code)
Duration (P50, P95, P99 latency)
Saturation (CPU/memory per pod, queue depth)

PromQL snippets

# Rate
sum(rate(http_requests_total[1m])) by (service, endpoint)

# Error ratio
sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m]))

# P99 latency
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

Dashboard 2: Trace Explorer

Top 10 slowest traces in last hour
Trace heatmap (duration vs. timestamp)
Service dependency graph (from Tempo service graph)
High-error traces panel (filter by status.error=true)

Dashboard 3: The "Burndown" Chart

Error budget remaining (daily trend line)
SLO burn rate (1h, 6h, 24h windows)
Multi-burn alert status (green/yellow/red)
Top offending services by error budget consumption

Why this works: On-call opens Dashboard 1 → sees elevated latency → clicks a trace in Dashboard 2 → finds slow database query → checks Dashboard 3 to decide if paging SREs is urgent.

Final checklist for production readiness

Before you sleep soundly:

Ingestion testing: curl a test span/metric/log through the Collector.
Retention: Set Mimir 30d, Tempo 14d, Loki 30d (adjust to compliance).
Auth: Add Grafana OAuth (Google/GitHub) and basic auth for Mimir/Loki ingesters.
Backups: Object storage (MinIO/S3) should have versioning enabled.
Alert testing: Silence a service, verify PagerDuty gets the page.
Runbook: Link each alert to a Confluence doc (e.g., "ErrorBudgetExhausted → https://wiki/runbooks/slo").

What’s next? Add OpenTelemetry for your database (PostgreSQL, Redis, MongoDB) using OTel collector receivers. Or add synthetic monitoring with Blackbox exporter.

You now have the same stack that cost my clients $0/month (excluding storage) instead of $15k/month for Datadog. Ship it.

From GitHub Actions to ArgoCD: A Complete GitOps Migration Guide

varun varde — Tue, 07 Apr 2026 13:33:40 +0000

I've done this migration three times now. Each time, I learn something new about what NOT to do. This guide is the documentation I wish I'd had the first time: the complete migration from a multi-stage GitHub Actions deployment pipeline to a fully declarative GitOps workflow with ArgoCD including the parts every other tutorial glosses over.

What you'll build: Production ready ArgoCD installation, ApplicationSet for multi cluster management, Image Updater for automated rollouts, RBAC configuration for multi-team environments, Rollback procedures that work under pressure.

Prerequisites: Kubernetes cluster (1.24+), kubectl, Helm 3, existing GitHub Actions CI pipeline. Estimated time: 4-6 hours for learning setup (30 days for production migration).

Phase 1: Assessment — Mapping Your Existing GitHub Actions Pipeline

Every successful migration begins with clarity. Before introducing GitOps, the existing CI/CD topology must be dissected—methodically, not casually.

Start by inventorying workflows

# .github/workflows/deploy.yml
name: Deploy Service

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: docker build -t myapp .

  deploy:
    needs: build
    steps:
      - run: kubectl apply -f k8s/

This pipeline reveals implicit assumptions:

Deployment is tightly coupled with CI
Kubernetes manifests are applied imperatively
No state reconciliation exists

Map each component

Component	Purpose
Build Step	Artifact creation
Deploy Step	Cluster mutation
Secrets	Runtime configuration

The goal is not replication but transformation. GitOps demands declarative intent, not procedural execution.

Phase 2: ArgoCD Installation (Production-Grade Helm Values)

A default installation is rarely sufficient. Production demands rigor secure ingress, RBAC, and resource constraints.

Install ArgoCD using Helm

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd --create-namespace -f values.yaml

A hardened values.yaml might resemble

server:
  service:
    type: ClusterIP

  ingress:
    enabled: true
    hosts:
      - argocd.example.com

configs:
  params:
    server.insecure: false

  cm:
    url: https://argocd.example.com

redis:
  enabled: true

controller:
  resources:
    limits:
      cpu: 500m
      memory: 512Mi

Security is not ornamental. It is foundational. Misconfigured access at this stage invites future instability.

Phase 3: Structuring Your GitOps Repository

Git becomes the single source of truth. Structure, therefore, becomes paramount.

A canonical layout

gitops-repo/
├── apps/
│   ├── payments/
│   ├── auth/
├── environments/
│   ├── dev/
│   ├── staging/
│   └── prod/
└── infrastructure/
    ├── ingress/
    └── monitoring/

Each directory encapsulates intent.

Example application manifest

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-service
spec:
  source:
    repoURL: https://github.com/org/gitops-repo
    path: apps/payments
    targetRevision: HEAD
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Declarative. Predictable. Auditable.

Phase 4: ApplicationSet Configuration for Multi-Team

At scale, manually defining applications becomes untenable. ApplicationSet introduces dynamic orchestration.

A Git generator example

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-apps
spec:
  generators:
    - git:
        repoURL: https://github.com/org/gitops-repo
        revision: HEAD
        directories:
          - path: apps/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/org/gitops-repo
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'

Teams simply add a folder. The system reacts autonomously.

This is where GitOps begins to feel almost sentient responsive, adaptive, yet deterministic.

Phase 5: Migrating Your First Service (with Rollback Plan)

Start small. Select a non-critical service. Minimize risk while maximizing insight.

Step 1: Remove Deploy Step from CI

# Keep build, remove kubectl apply
- run: docker build -t myapp .

Step 2: Push Kubernetes Manifests to GitOps Repo

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 2

Step 3: Let ArgoCD Sync

ArgoCD reconciles desired state with actual state.

Rollback Strategy

Rollback becomes trivial

git revert <commit-hash>
git push origin main

No manual intervention. No frantic patching. Just version control.

Phase 6: Image Updater for CI Integration

CI still builds images. But deployment is decoupled.

ArgoCD Image Updater bridges the gap

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: myapp=repo/myapp
    argocd-image-updater.argoproj.io/update-strategy: latest

Pipeline pushes image

docker push repo/myapp:1.2.3

Image Updater modifies Git

image: repo/myapp:1.2.3

ArgoCD syncs automatically.

Elegant. Autonomous.

Phase 7: RBAC and Multi-Tenant Configuration

As teams proliferate, governance becomes essential.

Define roles

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
data:
  policy.csv: |
    p, role:dev, applications, get, */*, allow
    p, role:dev, applications, sync, */*, allow
    g, team-dev, role:dev

Namespace isolation

spec:
  destination:
    namespace: team-a

Each team operates within boundaries autonomous yet controlled.

Phase 8: Observability for Your GitOps Pipelines

Visibility transforms guesswork into certainty.

Enable metrics

controller:
  metrics:
    enabled: true

Scrape with Prometheus

- job_name: 'argocd'
  static_configs:
    - targets: ['argocd-metrics:8082']

Key metrics

Sync status
Reconciliation duration
Drift detection

Logs provide narrative

kubectl logs deployment/argocd-application-controller

Without observability, GitOps becomes a black box. With it, a glass box.

The 5 Things No Tutorial Mentions (Learned the Hard Way)

1. Git Becomes Your Bottleneck

Frequent commits can overwhelm repositories. Optimize branching strategies. Avoid unnecessary churn.

2. Drift Happens More Than Expected

Manual changes in clusters still occur. ArgoCD corrects them—but not always instantly. Expect transient inconsistencies.

3. Secrets Are a Persistent Challenge

Plain YAML is insufficient. Integrate tools like sealed secrets or external secret managers.

4. ApplicationSet Can Spiral into Complexity

Dynamic generation is powerful—but dangerous. Poor templates create cascading misconfigurations.

5. Human Behavior Is the Hardest Problem

Engineers accustomed to imperative deployments resist change. Training is not optional. It is essential.

Migrating to ArgoCD is not merely a tooling shift. It is a philosophical transition—from imperative control to declarative intent.

The rewards are substantial. Stability. Traceability. Confidence.

But the journey demands discipline. And a willingness to relinquish old habits in favor of something far more resilient.

Zero Downtime Cloud Migration: The 6-Phase Playbook

varun varde — Thu, 26 Mar 2026 08:27:40 +0000

Phase 1 is never 'lift and shift.' Here's the framework that keeps production stable throughout the entire move.

After leading migrations for organizations ranging from 200 to 4,000 engineers, I've distilled it into 6 phases that keep production alive and teams sane.

Phase 1: Discovery & Dependency Mapping
Before touching a single VM, audit everything.

Inventory all services: apps, databases, middleware, integrations
Map inter service dependencies including the undocumented ones (ask the engineers who've been there longest)
Tag everything by criticality, data sensitivity, migration complexity
Identify the "spiders in the web" services everything else depends on

Tools: AWS Application Discovery Service, Cloudamize, or a structured spreadsheet + interviews.

Phase 2: Cloud Foundation & Landing Zone
Build the platform before you migrate anything.

Set up your VPC architecture (hub spoke or flat decide now)
Implement IAM roles, SCPs, and guardrails
Deploy centralized logging, monitoring, and alerting
Establish your IaC standard (Terraform modules, Pulumi stacks — standardize before day one)
Set cost budgets and alerts BEFORE anything runs
Never skip this. Teams that migrate first and "sort out governance later" always regret it.

Phase 3: Pilot Migration (Noncritical workloads)
Start small. Build confidence.

Choose a non-critical, low traffic service
Run full migration lifecycle: move → validate → monitor → optimize
Document everything. This becomes your runbook for every subsequent migration
Identify gaps in tooling and process while the stakes are low

Phase 4: Wave Based Migration
Organize workloads into migration waves by complexity:

Wave 1: Stateless apps (easiest)
Wave 2: Stateful apps with managed DB alternatives
Wave 3: Complex integrations and legacy services
Wave 4 (optional): Services that require refactoring before migration

Run each wave over 2–4-week sprints. Include a 2-week stabilization window after each wave before proceeding.

Phase 5: Cutover & Traffic Management
Zero-downtime means dual running, not big-bang.

Use feature flags and DNS-based traffic shifting (Route 53 weighted routing or equivalent)
Implement a circuit breaker to instantly roll back traffic if error rates spike
Keep on prem running in parallel for 30–60 days post cutover
Don't decommission anything until you've run at least one full billing cycle on cloud

Phase 6: Optimize & Decommission
The migration isn't over when the apps are running.

Right-size instances based on real usage data (wait at least 2 weeks)
Implement autoscaling where you were running fixed capacity
Set up FinOps dashboards cloud spend will be surprising at first
Decommission on prem systematically, not in a rush
Run a migration retrospective with every team involved

The Rule I Never Break
No service goes to production without observability in place (logs, metrics, traces), a documented runbook, an on-call rotation assigned, and a tested rollback procedure.

Save for your next migration kickoff

Terraform Modules That Actually Scale: Patterns from 20 Years

varun varde — Wed, 18 Mar 2026 08:47:03 +0000

Provisioning was once an artisanal craft defined by shell scripts and fragile golden images. We've since pivoted to the declarative, idempotent paradigm of Terraform. But scaling isn't just about resource count it’s about managing the cognitive load of massive environments. At some point, the question stops being how do I write HCL? and becomes how do I stop this complexity from collapsing under its own weight?

The Monolith Trap: Why Large State Files Kill Velocity

The Mega State is the ultimate architectural debt. Dumping your VPC, EKS cluster, and RDS instances into one root module creates a catastrophic blast radius. One syntax error in a security group shouldn't lock the state file for your entire production database. Scaling demands decoupling. Infrastructure components must be treated as independent lifecycles. Networking is slow and steady app pods are ephemeral. Isolate them to minimize plan times and risk.

Strict Typing and Input Orthogonality

Ditch type = any. It’s lazy and dangerous. High utility modules require strict object validation to fail fast. If the calling code sends a malformed schema, the plan should die immediately not thirty minutes into a deployment.

variable "cluster_config" {
  description = "EKS spec"
  type = object({
    version    = string
    vpc_id     = string
    subnet_ids = list(string)
    node_groups = map(object({
      instance_types = list(string)
      capacity_type  = string
    }))
  })

  validation {
    condition     = contains(["1.28", "1.29", "1.30"], var.cluster_config.version)
    error_message = "K8s version restricted by organizational compliance."
  }
}

Compositional Strategy: Favoring Layering over Nesting

Nesting modules inside modules leads to Prop Drilling a nightmare where a variable is passed through five layers of code just to reach a single resource. It's brittle.

Layering is the professional choice. Keep modules flat and use a Composition Root to stitch outputs to inputs. This keeps the logic readable and the dependencies explicit.

Cross Account Orchestration via Provider Aliasing

Enterprise scale means multiple regions and hundreds of AWS accounts. Never hardcode provider blocks inside a module; it makes them non portable. Use Provider Aliases. This allows you to instantiate the same module across different regions within a single run.

# Instance in US-East
module "s3_east" {
  source = "./modules/s3"
  providers = {
    aws = aws.us-east-1
  }
}

# Instance in US-West
module "s3_west" {
  source = "./modules/s3"
  providers = {
    aws = aws.us-west-2
  }
}

Immutable Versioning and Private Registries

Local file paths are for hobbyists. In production, referencing a local path means every save on your laptop could theoretically break a CI/CD pipeline. Scaling requires immutable versioning. Use Git tags or a Private Registry. It’s the only way to ensure Production stays on v1.2.0 while Dev experiments with v2.0.0 beta.

Automated Validation: Terratest and OPA Integration

If you aren't testing, you're guessing. For core modules, Terratest is the standard. It spins up real resources, pings them to ensure they work, and tears them down. Couple this with Open Policy Agent (OPA) to programmatically block anyone from creating an unencrypted volume or a public S3 bucket before the code even leaves the PR.

Day 2 Operations: Refactoring with moved Blocks

Refactoring used to mean terraform state mv commands that risked corrupting the remote backend. Now, we have the moved block. It allows you to move resources into modules without triggering a "destroy and recreate" cycle.

moved {
  from = aws_instance.legacy_app
  to   = module.app_cluster.aws_instance.this[0]
}

This is how you turn a tangled mess of HCL into a robust ecosystem. It’s about predictability, isolation, and strict interfaces.