DEV Community: Alok Ranjan Daftuar

Why Agent Infrastructure Is Its Own Discipline

Alok Ranjan Daftuar — Fri, 18 Sep 2026 05:39:20 +0000

Ask a platform team how they're going to run their first production agent and you'll get a confident answer within ten seconds: containerize it, put it behind an ingress, wire up a Horizontal Pod Autoscaler, done. It's the same playbook that's shipped every stateless service for the last decade, and there's no obvious reason an agent should be different. It accepts a request. It returns a response. It's just a container.

That answer is wrong — and it's wrong in a way that doesn't show up in a demo. It shows up three weeks into production, when a pod gets killed mid-reasoning because a liveness probe decided a 40-second tool call was a hang, or when the autoscaler adds five replicas because CPU spiked on a single request calling six tools in sequence, or when two "sessions" turn out to share a K8s Service without anyone having reasoned about what that means for isolation.

The shape of an agent request vs. a microservice request

A typical microservice request has a shape platform engineers have spent fifteen years optimizing around: bounded latency, a single unit of compute per request, statelessness between requests, and a clear success/failure signal at the HTTP layer. An agent request breaks all four at once.

Standard microservice request:
  client -> service -> [DB/cache lookup] -> response
  Duration: milliseconds to low seconds
  Compute: roughly constant per request
  State: none carried between requests
  Outcome signal: HTTP status code

Agent request:
  client -> agent -> [reason] -> tool call 1 -> [reason] -> tool call 2
        -> [reason] -> tool call N -> [reason] -> response
  Duration: seconds to minutes, highly variable
  Compute: proportional to reasoning depth and tool fan-out
  State: conversation/task context carried across the entire chain
  Outcome signal: HTTP 200 with a semantically wrong answer is common

That last line is the one platform teams underestimate most. An agent that loops on a tool, calls the wrong one, or returns a plausible-but-incorrect result will still return a healthy status code. The infrastructure layer has no way to distinguish a correct run from a confidently wrong one.

Where the standard playbook breaks

Microservice assumption	Agent reality	Practical implication
Liveness = "process is alive and responsive"	A live agent can be legitimately unresponsive for 30–90 seconds mid-tool-call	Naive liveness probes kill healthy pods mid-reasoning
CPU/memory tracks load	Load tracks reasoning depth and tool fan-out, not CPU	HPA on CPU/memory over- or under-scales unpredictably
Requests are independent	A single task often spans multiple round trips carrying shared context	Session/state handling needs an explicit design, not an assumption
One replica serves any request	Tool credentials and context may be tenant-specific	Routing and isolation boundaries need to be architectural, not incidental
Timeout = failure	Timeout at 30s might just mean the agent is still reasoning	Timeout budgets have to be set per tool-call chain, not per request

The liveness probe problem, concretely

With periodSeconds: 10 and failureThreshold: 3, a standard probe kills a pod if /health doesn't respond within roughly 30 seconds — which is an entirely normal duration for a reasoning step that includes a tool call to a slow downstream API.

The fix isn't a bigger failureThreshold. It's separating "the process is alive" from "the process is making forward progress":

livenessProbe:
  httpGet:
    path: /health          # answers: is the process itself alive?
    port: 8080
  periodSeconds: 15
  timeoutSeconds: 5
  failureThreshold: 3      # ~45s of true unresponsiveness before restart

readinessProbe:
  httpGet:
    path: /ready            # answers: can this pod accept new work right now?
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

/health should do nothing more than confirm the process's event loop is running — never block on the status of an in-progress tool call. /ready reflects capacity: a pod mid-reasoning can report itself not-ready for new work without being treated as dead. For a request/response service these two questions have the same answer. For an agent, they routinely don't.

What changed in the protocol layer

A large part of the awkwardness in running MCP-based agents on Kubernetes came from the original protocol design, which required persistent, pinned sessions between a client and a specific server instance — the opposite of what horizontally scaled infrastructure wants. That constraint forced teams into sticky routing and shared session stores just to keep a conversation coherent across requests.

The July 2026 MCP specification revision removed the protocol-level session entirely. Any request can now land on any server instance, and applications that need to carry state across calls do it the way HTTP APIs always have — by minting an explicit handle passed back as an ordinary argument, rather than relying on the transport to remember. That single change removes an entire category of infrastructure workaround (sticky routing, pinned sessions, shared session stores) that used to be treated as unavoidable.

The second shift is A2A (Agent-to-Agent protocol), which reached v1.0 in early 2026. Where MCP governs how an agent talks to tools and data sources, A2A governs how agents talk to each other — and that distinction matters for infrastructure design. A multi-agent system where agents call each other over A2A has different routing, identity, and isolation requirements than one where a single orchestrator calls tools over MCP. Both patterns are in production today.

Why the cost of getting this wrong compounds quietly

A liveness probe that kills healthy pods doesn't fail loudly — it shows up as an elevated error rate attributed to "model flakiness" or "the tool API being unreliable," and teams spend weeks tuning retry logic against a problem that's actually a probe misconfiguration one layer down. An autoscaler tuned on the wrong signal doesn't fail either — it just runs 30% more replicas than the workload needs, indefinitely, because nobody has a reason to suspect the scaling metric itself.

Getting the infrastructure layer right doesn't guarantee correct agent behavior, but getting it wrong guarantees you can't tell the difference between an agent that's actually failing and one that's simply being run on infrastructure that wasn't built for it.

Read the Full Article

The summary covers the core argument. The full article goes deeper on:

The full breakdown of each assumption failure — autoscaling signal, session and state design, isolation boundaries, and timeout budget — with the specific production failure mode each one produces
Why EKS and AKS don't solve the shape problem by default, and how the implementations diverge (IRSA vs. workload identity federation, ALB Ingress vs. Application Gateway) even when the underlying design goal is identical
The complete naive vs. corrected deployment YAML side-by-side, with the exact reasoning behind each change

👉 Why Agent Infrastructure Is Its Own Discipline — Full Article

VPC Peering vs Transit Gateway: Choosing the Right AWS Inter-VPC Connectivity Model

Alok Ranjan Daftuar — Tue, 15 Sep 2026 11:35:40 +0000

The default assumption most teams start with is that VPC peering is the "simple" option and Transit Gateway is the "enterprise" option you graduate into once you're big enough. That framing is wrong often enough to cause real architectural pain. The actual decision isn't about company size — it's about topology shape.

Before comparing features, answer one question: does your connectivity requirement look like a mesh or a hub-and-spoke? That single test resolves most of the debate.

The math that kills peering at scale

A peering connection is non-transitive — A cannot reach C through B even if both connections exist. For N VPCs in a full mesh, the connections required are N * (N - 1) / 2:

VPCs	Peering connections	TGW attachments
3	3	3
5	10	5
10	45	10
20	190	20

At 10 VPCs you're managing 45 separate connections, each with its own route table entries on both sides. Transit Gateway needs exactly N attachments regardless of how many VPCs you have.

Setting up VPC Peering in Terraform

resource "aws_vpc_peering_connection" "app_to_shared" {
  vpc_id      = aws_vpc.app.id
  peer_vpc_id = aws_vpc.shared_services.id
  auto_accept = true
}

resource "aws_route" "app_to_shared" {
  route_table_id            = aws_route_table.app_private.id
  destination_cidr_block    = aws_vpc.shared_services.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.app_to_shared.id
}

resource "aws_route" "shared_to_app" {
  route_table_id            = aws_route_table.shared_private.id
  destination_cidr_block    = aws_vpc.app.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.app_to_shared.id
}

Both directions need an explicit route — peering doesn't propagate routes automatically. Every new VPC added to a peered mesh means touching route tables on every existing member.

Setting up Transit Gateway with segmented routing

resource "aws_ec2_transit_gateway" "main" {
  description                     = "tgw-central-networking"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"
}

resource "aws_ec2_transit_gateway_vpc_attachment" "app" {
  subnet_ids         = [aws_subnet.app_private_a.id, aws_subnet.app_private_b.id]
  transit_gateway_id = aws_ec2_transit_gateway.main.id
  vpc_id             = aws_vpc.app.id
}

The disable pair on both defaults is deliberate — it enables route table segmentation instead of every attachment automatically seeing every other attachment.

Route table segmentation: the feature peering can't replicate

A single Transit Gateway can have multiple route tables. Each attachment associates with one, giving you actual network segmentation without touching security groups or NACLs:

TGW Route Table: "production"
├── app-vpc attachment    (associated)
├── shared-vpc attachment (associated, propagated)
└── sandbox-vpc attachment (NOT associated — isolated by design)

TGW Route Table: "sandbox"
├── sandbox-vpc attachment (associated)
└── shared-vpc attachment  (associated, propagated)

This is the pattern most multi-account AWS Organizations setups converge on: one Transit Gateway per region, a small number of route tables representing trust boundaries, and VPCs associated into the appropriate table.

Cross-account via AWS RAM

Transit Gateway solves the multi-account case cleanly through Resource Access Manager — one account owns the TGW, shares it via RAM, and spoke accounts create their own attachments:

resource "aws_ram_resource_share" "tgw_share" {
  name                      = "tgw-network-share"
  allow_external_principals = false
}

resource "aws_ram_resource_association" "tgw" {
  resource_arn       = aws_ec2_transit_gateway.main.arn
  resource_share_arn = aws_ram_resource_share.tgw_share.arn
}

resource "aws_ram_principal_association" "spoke_account" {
  principal          = "222233334444"
  resource_share_arn = aws_ram_resource_share.tgw_share.arn
}

Peering works cross-account too, but the N(N-1)/2 problem gets worse because you're also managing acceptance workflows and cross-account IAM permissions for every connection.

DNS and security group gotchas

Routing connectivity doesn't automatically mean DNS works across it. For peering, DNS resolution support must be enabled explicitly on both sides — without it, private hosted zone records resolve to public IPs or fail entirely, which looks identical to a routing problem.

Security group referencing across peering connections is also limited — most cross-account or cross-region setups fall back to CIDR-based rules rather than SG-to-SG references. Transit Gateway doesn't change this constraint.

Troubleshooting in priority order

When "VPC-A can't reach VPC-B" comes in:

Route table on both sides — both directions need explicit routes
Connection/attachment state — pending-acceptance on peering, non-available TGW attachment
Security group and NACL on both ends
DNS resolution settings — easy to mistake for a routing failure
TGW route table association — an attachment not associated with the expected route table silently fails

Read the Full Article

The full post covers everything above in depth, plus:

The DNS resolution strategy for hub-and-spoke setups using Route 53 Resolver endpoints — why it needs to be planned alongside routing, not retrofitted
The full side-by-side comparison table across all dimensions (routing model, cost, cross-region, segmentation, best fit)
The migration path from peering to Transit Gateway — running both in parallel, using most-specific-match routing for a controlled cutover without an all-or-nothing switch
When peering is genuinely the right answer and why you shouldn't add TGW complexity to a two-VPC problem

👉 VPC Peering vs Transit Gateway — Full Article

CI to GitOps Handoff: Argo Image Updater vs. CI-Writes-the-Commit vs. Kargo

Alok Ranjan Daftuar — Thu, 10 Sep 2026 09:23:01 +0000

ArgoCD only reconciles what's in Git. A CI pipeline builds an image, pushes it to a registry — and then what? Something has to turn "a new image exists" into "a commit exists in the GitOps repo referencing it." That handoff is where a surprising number of GitOps implementations quietly go wrong: either it's done manually, or it's automated in a way that reintroduces the exact imperative-deploy risk GitOps was supposed to remove.

Three approaches cover almost every real setup. The choice isn't just operational preference — it interacts directly with your repo structure and how much promotion-stage enforcement you actually need.

The structural difference in one picture

Approach 1 — Argo Image Updater
  CI: build → push image ──────────────► registry
                                              │ (polled every 1-2 min)
                                              ▼
                                    Image Updater controller
                                              │ writes tag
                                              ▼
                                     GitOps repo commit ──► ArgoCD syncs

Approach 2 — CI writes the commit
  CI: build → push image ──► registry
       │
       └── same workflow, next job: bump tag, commit, push ──► GitOps repo ──► ArgoCD syncs

Approach 3 — Kargo
  CI: build → push image ──► registry
                                 │
                                 ▼
                            Kargo Warehouse (watches registry)
                                 │
                                 ▼
                          Stage: staging ──(verify)──► Stage: prod
                                 │                          │
                                 ▼                          ▼
                          GitOps repo commit          GitOps repo commit
                          (staging path)               (prod path, only if
                                                         staging verified)

Approaches 1 and 2 go straight from "new image" to "commit" with no concept of one environment gating another. Kargo inserts an explicit, enforced step between them. Everything below is detail on why you'd pick one over another.

Argo Image Updater

Image Updater runs alongside ArgoCD, polls registries for new tags matching a pattern, and writes the updated tag back into the GitOps repo. CI's job stays exactly "build, test, push image" — no GitOps-repo awareness required.

Configuration lives as annotations on the Application itself:

annotations:
  argocd-image-updater.argoproj.io/image-list: checkout=111122223333.dkr.ecr.us-east-1.amazonaws.com/checkout-service
  argocd-image-updater.argoproj.io/checkout.update-strategy: semver
  argocd-image-updater.argoproj.io/checkout.allow-tags: 'regexp:^v[0-9]+\.[0-9]+\.[0-9]+$'
  argocd-image-updater.argoproj.io/write-back-method: git
  argocd-image-updater.argoproj.io/git-branch: main

The write-back-method: git setting is the important one. The alternative (argocd) writes to the Application object only, meaning the running state and the Git repo can silently diverge — undermining the entire premise of GitOps as the source of truth. If you use Image Updater, git write-back is the only mode consistent with a real GitOps model.

Trade-off: polling-based (1-2 minute delay), and no native concept of promotion stages. Getting a build to flow dev → staging → prod with any gate means bolting that logic on separately — which is exactly the gap Kargo exists to close.

CI writes the manifest commit

No separate controller. The CI job itself pushes the commit after a successful build. The auth chain here deserves the same scrutiny as cloud credentials — a long-lived PAT with write access to the GitOps repo is the wrong answer for the same reasons a long-lived AWS access key is wrong.

The right pattern: a GitHub App scoped to exactly the one GitOps repo, with an installation token minted per workflow run:

jobs:
  update-gitops-repo:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Generate GitHub App token for GitOps repo
        id: app-token
        uses: actions/create-github-app-token@v1
        with:
          app-id: ${{ vars.GITOPS_BOT_APP_ID }}
          private-key: ${{ secrets.GITOPS_BOT_PRIVATE_KEY }}
          owner: example-org
          repositories: checkout-team-gitops
      - uses: actions/checkout@v6
        with:
          repository: example-org/checkout-team-gitops
          token: ${{ steps.app-token.outputs.token }}
      - name: Bump image tag
        run: yq -i '.image.tag = "${{ github.sha }}"' apps/checkout-service/prod/values.yaml
      - name: Commit and push
        run: |
          git config user.name "gitops-bot"
          git config user.email "gitops-bot@example.org"
          git commit -am "checkout-service: bump to ${{ github.sha }}"
          git push

The token is scoped to one installation (one repo) and expires roughly an hour after issuance — no long-lived credential sitting in a secrets store.

Trade-off: no delay, no extra controller. But CI now has push access to the thing that controls production. A GitHub App token scoped org-wide instead of to one repo turns a compromised CI secret from a one-service incident into an org-wide one. The scoping isn't optional hardening — it's the difference between those two outcomes.

Kargo

Image Updater and CI-writes-the-commit both update one Application at a time with no native concept of "this build must pass staging before prod." Kargo models that as a first-class API.

Two core resources: a Warehouse (watches for new artifacts, like Image Updater) and a Stage (one step in a promotion pipeline). The key field:

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: prod
  namespace: checkout-team
spec:
  requestedFreight:
    - origin:
        kind: Warehouse
        name: checkout-service
      sources:
        stages: [staging]   # prod can only promote freight that passed through staging

sources: stages: [staging] means prod is structurally incapable of promoting an image that hasn't gone through staging — enforced by Kargo, not by convention. Verification gates (smoke tests, analysis runs) attach to each Stage, and Kargo tracks exactly which artifact made it through which Stages.

The distinction from bolting a manual approval onto Image Updater: the gate is a property of the Stage graph itself, checked by Kargo before it constructs a promotion for the next Stage. Not a step someone has to remember to run.

Trade-off: another controller with its own resource model on top of ArgoCD's. Earns its cost specifically when multi-stage promotion with gated, auditable progression is a real requirement — not as a default for every service.

Decision framework

Situation	Recommended approach
Single environment, or environments promote independently	Argo Image Updater — simplest, no CI changes needed
Simple promotion logic already visible in CI (merge to main = staging, tag = prod)	CI writes the commit — keeps logic in one place, no extra controller
Hard requirement that prod only receives artifacts that passed staging with automated verification	Kargo — this is the exact problem it's built to solve
Regulated workloads needing an auditable promotion trail as a first-class object	Kargo — Stage/Freight model gives you that, not something reconstructed from CI logs

These aren't mutually exclusive across a platform — a team running mostly independent services on Image Updater might still put its most compliance-sensitive service through Kargo specifically.

Read the Full Article

The summary covers the shape of each approach and the key trade-offs. The full article goes deeper on:

The complete Image Updater Application manifest — image-list, update-strategy, allow-tags, and write-back-method in context, not just the annotation snippet
The full two-job GitHub Actions workflow (build + update-gitops-repo), including the step that verifies the App token fails against repos it shouldn't reach — confirming blast radius, not just that it works
The complete Kargo Warehouse + Stage YAML including the verification block that wires an AnalysisTemplate smoke test to the staging Stage — the part that makes "staging verified" mean something concrete
Why a Matrix generator with a too-broad cluster label selector connects directly to the auth chain here: AppProject destinations as the backstop when the ApplicationSet generates more than intended
Article 4 preview: secrets management (Sealed Secrets, External Secrets Operator, Vault plugin) — what each actually protects against and what it doesn't

👉 CI to GitOps Handoff: Argo Image Updater vs. CI-Writes-the-Commit vs. Kargo — Full Article

Always Encrypt in Transit: The Gap Between TLS Everywhere and Actual Transport Security

Alok Ranjan Daftuar — Tue, 08 Sep 2026 07:26:21 +0000

"We have TLS everywhere." It appears in every architecture review and every compliance questionnaire. The gap is not in the intention — it is in the implementation. Most systems that claim TLS everywhere have TLS at the edge and plaintext everywhere else: the ingress controller to the pod is HTTP, pod-to-pod traffic is unencrypted, and the application-to-database connection string never had Encrypt=True or sslmode=require set. None of these gaps appear in architecture diagrams.

This post maps where TLS is actually absent, why the gaps exist, and the implementation path that closes them — without defaulting to "add a service mesh" as the answer to every transport security question.

The Four Layers Where TLS Is Actually Absent

Before solutions, the precise inventory of where plaintext exists in systems claiming TLS everywhere:

Scenario: "Secure" architecture
  ✅ HTTPS on the load balancer
  ✅ Private subnets, no public IPs
  ✅ Security groups restricting access
  ❌ Ingress Controller → Pod: plaintext HTTP
  ❌ Pod → Pod: plaintext HTTP/gRPC
  ❌ Application → RDS: plaintext TCP
  ❌ Cluster infrastructure certs: expire annually, no automated rotation

What "TLS everywhere" actually means in this architecture:
  TLS on the edge. Plaintext the rest of the way.

Layer 1 — Ingress-to-pod: nginx-ingress and Traefik terminate TLS at the edge and forward plain HTTP to backend pods by default. The gap is invisible in architecture diagrams.

Layer 2 — Pod-to-pod: Kubernetes does not encrypt data plane traffic. A compromised pod on a shared node can observe plaintext traffic from other pods on the same node — the VPC boundary is not the relevant perimeter here, the pod boundary is.

Layer 3 — Application-to-database: Enabling TLS on RDS makes the database capable of TLS connections. It does not enforce them. A connection string without sslmode=require or Encrypt=True connects over plaintext regardless.

Layer 4 — Cluster infrastructure certs: kubeadm cluster CA certificates expire after one year by default with no automated renewal. A missed rotation takes down the entire cluster — not an application outage, a cluster-level failure where kubectl stops working entirely.

Termination Architecture: Where the Decision Gets Made

Where TLS terminates determines which traffic is encrypted. Three patterns:

Edge termination only — TLS terminates at the ingress controller, all internal traffic is plaintext. Legitimate for single-tenant clusters with no regulated data. Not legitimate to call "TLS everywhere."

Re-encryption — the ingress controller terminates external TLS and re-encrypts before forwarding to the pod. Closes the ingress-to-pod gap without a service mesh:

# NGINX Ingress: re-encrypt to backend over TLS
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-ingress
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
  - hosts: [api.yourdomain.com]
    secretName: api-tls-secret
  rules:
  - host: api.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 8443

End-to-end mTLS via service mesh — the mesh intercepts all pod-to-pod traffic and wraps it in mTLS. Strongest posture, highest operational overhead. Justified in regulated multi-tenant clusters; disproportionate for smaller service estates.

cert-manager: The Implementation That Actually Automates It

cert-manager handles issuance, renewal, and storage in Kubernetes Secrets with zero manual steps in the rotation path. One ClusterIssuer and one Certificate resource per service:

# ClusterIssuer: Let's Encrypt production
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@yourdomain.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx
---
# Certificate: 90-day validity, auto-renewed at 60 days remaining
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-certificate
  namespace: production
spec:
  secretName: api-tls-secret
  duration: 2160h
  renewBefore: 720h
  dnsNames:
  - api.yourdomain.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

For internal service-to-service certificates, use a self-signed CA issuer with 24-hour validity — cert-manager makes short-lived internal certs operationally trivial, and a 24-hour certificate has a dramatically smaller blast radius than a one-year certificate if the key is compromised:

# Internal service certificate — 24h validity, rotated automatically
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: order-service-cert
  namespace: production
spec:
  secretName: order-service-tls
  duration: 24h
  renewBefore: 8h
  dnsNames:
  - order-service.production.svc.cluster.local
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer

Certificate Lifecycle: Where Security Controls Become Outages

The failure mode is consistent: a certificate is issued, configured, and forgotten. The alert that was supposed to fire before expiry never got configured, or fired into a channel nobody watches. The certificate expires. The service goes down. The postmortem recommends "better monitoring" — until the same thing happens eighteen months later with a different certificate.

Alert on time-to-expiry, not on expiry. A Prometheus alert that fires at 15 days remaining gives the team time to investigate before the outage. An alert that fires when the certificate has already expired is a notification of an ongoing incident:

groups:
- name: certificate-expiry
  rules:
  - alert: CertificateExpiringIn15Days
    expr: |
      certmanager_certificate_expiration_timestamp_seconds
        - time() < (15 * 24 * 3600)
    for: 1h
    labels:
      severity: warning
    annotations:
      summary: "Certificate expiring soon"
  - alert: CertificateExpired
    expr: |
      certmanager_certificate_expiration_timestamp_seconds - time() < 0
    labels:
      severity: critical
    annotations:
      summary: "Certificate has EXPIRED"

Both alerts are required. Most teams have only the expired alert — which is a notification of an ongoing incident, not a prevention.

mTLS Without a Service Mesh

For fewer than twenty services, cert-manager internal CA plus application-level TLS provides mutual authentication without sidecar injection or a mesh control plane. The application handles the TLS handshake directly using certificates cert-manager issues and rotates automatically:

// .NET: present a client certificate and verify the server against the internal CA
var clientCert = X509Certificate2.CreateFromPemFile(clientCertPath, clientKeyPath);
var caCert     = new X509Certificate2(caCertPath);

var handler = new HttpClientHandler();
handler.ClientCertificates.Add(clientCert);
handler.ServerCertificateCustomValidationCallback = (_, cert, chain, _) =>
{
    chain!.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;
    chain.ChainPolicy.CustomTrustStore.Add(caCert);
    return chain.Build(cert!);
};

The certificates are mounted from Kubernetes Secrets that cert-manager manages. When cert-manager rotates the Secret, the volume mount is updated in place — no application restart required.

Read the Full Article

This summary covers the core gaps, termination architecture patterns, cert-manager setup, and the mTLS-without-mesh approach. The full article includes:

The complete database enforcement gap — rds.force_ssl, connection string Encrypt=True, and why enabling TLS on the DB is not the same as enforcing it
The VPC isolation misconception in full — why private subnets don't substitute for encryption
The "Kubernetes encrypts cluster traffic" misconception — what it actually encrypts vs. what it doesn't
Full decision framework mapping every traffic path to the right termination architecture
When regulated environments (PCI-DSS, HIPAA) make full end-to-end TLS non-negotiable
Eight key takeaways covering every layer of the in-transit security posture

👉 Always Encrypt in Transit: The Gap Between TLS Everywhere and Actual Transport Security — Full Article

AWS MGN Architecture: How Continuous Replication Actually Works

Alok Ranjan Daftuar — Thu, 03 Sep 2026 07:29:32 +0000

The first time I ran an AWS MGN migration, I trusted the console's green "Healthy" status without understanding what it actually meant. It took a stalled replication and a confusing lag metric mid-cutover to make me go back and learn the mechanics underneath the dashboard. This post is the breakdown I wish I'd had before that first wave.

MGN does one job: continuous, block-level replication of a source server's disks to a staging area in your target AWS account, so that when you cut over, you launch a fully synced, bootable EC2 instance instead of doing a one-time data copy and hoping nothing changed since. The distinction that matters is block-level, not file-level — MGN doesn't care about your filesystem or application state, which is what allows it to keep a target in near-real-time sync with a live, running source server throughout the migration.

The four components, in order

1. The replication agent — installed directly on the source server, hooks into the OS's disk I/O path at the kernel level. On Linux this means a kernel-level block device reader; on Windows, a filter driver. This low-level hook is why a kernel version mismatch on the source isn't a minor footnote — it's a direct threat to replication itself.

sudo python3 ./aws-replication-installer-init.py \
  --region ap-south-1 \
  --aws-access-key-id <access-key-id> \
  --aws-secret-access-key <secret-access-key> \
  --no-prompt

The --no-prompt flag matters for scripted wave installations — without it, the installer pauses for disk selection confirmation and silently stalls automation.

2. The staging area subnet — a designated subnet in the target AWS account where MGN provisions its replication infrastructure. Needs outbound connectivity and enough IOPS headroom on EBS to keep up with the source's write rate. Undersizing this subnet or placing it behind restrictive route tables is a common cause of replication that starts but never stabilizes.

3. The replication server and EBS volumes — for each source server, MGN provisions a temporary EC2 instance in the staging subnet whose only job is to receive the block-level stream and write it to EBS volumes that mirror the source's disk layout. These EBS volumes are the actual replica — they're what gets snapshotted at cutover.

The cost model people get wrong: you're paying for these replication servers and EBS volumes for the entire duration of the migration project, not just at cutover. A wave that sits in "replicating, not yet cut over" for six weeks accumulates staging infrastructure cost for all six weeks.

4. Continuous sync and the lag metric — once initial replication completes, MGN switches to continuous incremental sync: every write on the source is captured and streamed as it happens. The metric to watch is replication lag — how far behind the target EBS volumes are from the live source. Lag that climbs rather than staying flat is a leading indicator of a cutover that will either take much longer than expected or launch a target instance further behind the source than your rollback tolerance allows.

aws mgn describe-source-servers \
  --region ap-south-1 \
  --query 'items[*].{Server:sourceServerID, LagDuration:dataReplicationInfo.lagDuration, State:dataReplicationInfo.dataReplicationState}' \
  --output table

Use this as a scriptable pre-cutover gate rather than eyeballing a color indicator in the console.

What cutover actually triggers

Cutover is not a data operation — the data is already synced. What it actually does:

Takes a final snapshot of the target EBS volumes at the moment of cutover
Launches a new EC2 instance from that snapshot using the launch template you've configured
Runs the MGN post-launch conversion process — driver injection, network configuration, and OS-level changes needed to make a disk image that ran on a different hypervisor boot correctly under AWS's Nitro hypervisor

Step 3 is where kernel driver failures happen. The conversion process needs to inject or activate the correct network driver (ena) for the instance to have connectivity after launch. If the source server's kernel doesn't have the module available or the boot configuration doesn't reference it correctly, the instance can come up without network connectivity — or fail to boot entirely.

Launch settings most people under-invest in

Every source server has its own launch template in MGN. Worth setting deliberately before cutover, not left at defaults:

Target instance type — set explicitly for anything performance-sensitive rather than trusting MGN's right-size recommendation
Subnet and security groups — should be decided by your VPC CIDR plan, not chosen ad hoc at cutover time
IAM instance profile — doesn't carry over from the source server; must be set in the launch template explicitly
Test launch — MGN supports launching a test instance from current replication state without ending replication. Always run at least one test launch before the real cutover. Skipping this is the single most common way teams discover a launch-time problem during the actual cutover window instead of during a safe rehearsal

Replication settings that matter under load

Three settings worth understanding rather than leaving on defaults for source servers with high write throughput:

Bandwidth throttling — MGN doesn't cap replication bandwidth by default, which can compete with production traffic on a busy source server. Set a maximum throughput per source server for anything actively serving production load during the replication window
Staging volume type — the default EBS volume type for staging can itself become the bottleneck for high-IOPS sources, showing up as climbing lag even when network bandwidth is fine. Rule this in or out separately from the network-bandwidth cause when diagnosing a server that won't stabilize
Point-in-time (PIT) snapshots — MGN can retain a rolling window of snapshots of the staging volumes, giving you a recovery point earlier than "right now" if you need to launch from a known-good state. Configure this under Replication settings → Point-in-time snapshots in the MGN console before replication starts — snapshots only accumulate from the point the setting is enabled

Why continuous replication beats snapshot-and-copy

The snapshot approach has an unavoidable trade-off: the longer the gap between snapshot and cutover, the more source-side changes are missing from the target. Closing that gap means either accepting data loss or taking the source offline for the copy window.

MGN's continuous model removes that trade-off entirely. The target stays within seconds of the source right up until cutover, and the source never has to go offline for the migration itself — only for the brief cutover window when traffic is actually redirected. For anything with a real availability requirement, this is the entire reason to use MGN over a manual export/import process.

Read the Full Article

The summary covers the core mechanics. The full article goes deeper on:

The troubleshooting priority order for "these are definitely connected but nothing works" tickets — route tables, attachment state, security layers, DNS — and why checking in that sequence resolves the large majority of cases without reasoning about the full topology from scratch
The specific kernel driver failure mode that surfaces when migrating Azure-sourced Ubuntu images to AWS, why it manifests the way it does, and the fix
The full cost model for staging infrastructure across a multi-week wave, with the specific line items to flag before the wave kicks off
CIDR planning considerations for the staging subnet relative to the rest of the target VPC

👉 AWS MGN Architecture: How Continuous Replication Actually Works — Full Article

Enforcing Modular Monolith Boundaries in .NET: NDepend, Parallel Pipelines, and the Architecture That Holds

Alok Ranjan Daftuar — Mon, 31 Aug 2026 06:03:32 +0000

A modular monolith without enforcement is not an architecture — it is a monolith with good intentions.

The Problem

Most teams skip the modular monolith and jump straight to microservices. The ones that do attempt a modular monolith rely on convention — "don't cross module boundaries" — which fails the moment deadlines hit.

The difference between a well-structured modular monolith and a mess is whether boundaries are maintained by tooling or by convention.

The Solution Structure

Each module is a pair of .NET projects:

src/Modules/
  Orders/
    YourApp.Orders/              ← internal: domain, application, infrastructure
    YourApp.Orders.Contracts/    ← public: DTOs, interfaces, events
  Payments/
    YourApp.Payments/
    YourApp.Payments.Contracts/

The rule: modules may only reference each other's *.Contracts projects. The compiler enforces this physically — no project reference means no type access.

Four Layers of Enforcement

Compiler — project references prevent cross-module type access
NetArchTest — architecture tests fail the build on namespace-level violations
NDepend CQLinq — catches dependency cycles and coupling the compiler can't see
Quality Gates — block PRs that introduce new boundary violations

Module-Scoped Data

Each module owns a dedicated DbContext with a schema prefix (orders.*, payments.*). No module queries another module's tables.

Cross-Module Communication

Modules communicate via MediatR in-process events. Orders publishes OrderPlaced; Payments subscribes — without Orders knowing Payments exists.

This is also the extraction seam: when you eventually extract a module into a service, MediatR becomes a message broker. The event contract stays the same.

Parallel CI

strategy:
  matrix:
    module: [Orders, Payments, Inventory]
  fail-fast: false

Each module's tests run in parallel. CI time scales with the slowest module, not the total count.

The Extraction Path

When a module genuinely needs independence:

Add outbox table → publish to real broker
Replace MediatR handlers with broker consumers
Deploy module as separate service
Publish *.Contracts as NuGet package

The boundary was already clean. Extraction is a deployment change, not a redesign.

The full post covers NDepend CQLinq rule examples, Quality Gate configuration, GitHub Actions pipeline YAML, test isolation patterns, and a production checklist.

👉 Read the complete implementation guide

AWS VPC Networking Fundamentals: VPCs, Subnets, CIDR, Route Tables, IGW, and NAT Gateways

Alok Ranjan Daftuar — Fri, 28 Aug 2026 03:49:00 +0000

If you've provisioned a VPC from a Terraform module without fully internalising what each piece is doing, that's fine — right up until something breaks. An instance that should be reachable isn't. A private instance can't pull a package update. And you're left checking five different resources with no clear mental model of how they connect.

This post builds that mental model from the ground up. Not just definitions — the why behind each piece, so troubleshooting becomes deduction instead of guesswork.

CIDR math you actually need

A CIDR block is IP address / prefix length. The prefix length fixes the network portion; the remaining bits are your host space.

Formula: 2^(32 - prefix) = total addresses. AWS reserves 5 per subnet (network address, VPC router, DNS, reserved, broadcast).

CIDR	Total addresses	Usable
/16	65,536	65,531
/20	4,096	4,091
/24	256	251
/28	16	11

To reverse-engineer a prefix from a required host count: round up to the next power of two, subtract the exponent from 32. Need 300 hosts? Next power of two is 512 (2⁹), so prefix = 32 - 9 = /23. Run this before sizing any subnet that will host an autoscaling group or EKS node group.

Start with /16 for the VPC itself. VPC CIDR is difficult to resize after the fact — once you have subnets, peering connections, or Transit Gateway attachments built against it, renumbering becomes a migration project. /16 costs nothing up front and avoids that corner.

Subnet allocation: carving up the VPC

A practical three-AZ production layout from 10.0.0.0/16:

Tier	AZ-a	AZ-b	AZ-c	Size	Typical use
Public	10.0.0.0/24	10.0.1.0/24	10.0.2.0/24	/24	ALB, NAT gateway, bastion
Private/app	10.0.16.0/20	10.0.32.0/20	10.0.48.0/20	/20	EKS nodes, ECS, EC2
Data	10.0.64.0/24	10.0.65.0/24	10.0.66.0/24	/24	RDS, ElastiCache
Reserved	10.0.128.0/17			/17	Future tiers, Transit Gateway, VPN

The jump from /24 in the public tier to /20 in the app tier is intentional. ALBs and NAT gateways consume very few IPs; the app tier is where consumption scales with autoscaling groups, rolling deployments, and pod density.

For EKS specifically: with the VPC CNI, every pod can consume an ENI-backed IP. IP exhaustion is one of the most common EKS production incidents. /20 per AZ for worker subnets is the standard starting point.

The deliberate gaps between tiers (0–2, then 16–48, then 64–66) leave room to insert new tiers later without renumbering anything already deployed.

Route tables: the actual decision maker

A subnet is "public" or "private" because of its route table — not any inherent property of the subnet itself. The table is a list of destination → target rules evaluated by most specific match.

Every route table gets an implicit local route for the full VPC CIDR — this can't be removed, and it's what lets every subnet reach every other subnet inside the VPC by default.

A public subnet route table in Terraform:

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
  tags   = { Name = "rtb-public", Tier = "public" }
}

resource "aws_route" "public_internet" {
  route_table_id         = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.main.id
}

resource "aws_route_table_association" "public_a" {
  subnet_id      = aws_subnet.public_az_a.id
  route_table_id = aws_route_table.public.id
}

Creating the route table does nothing on its own — the association step is what binds it to a subnet and makes routing take effect.

Internet Gateway and the three conditions for inbound access

An IGW is horizontally scaled, redundant, and AZ-agnostic — one per VPC, no capacity to configure. It does two things: 1:1 NAT between public and private IPs (the public IP mapping lives at the IGW, not on the instance — which is why ip addr on an EC2 instance never shows its public IP), and serves as a route table target.

All three of these must be true simultaneously for inbound internet access to work:

The instance has a public or Elastic IP on its ENI.
The subnet's route table has 0.0.0.0/0 → IGW.
Both the security group and the NACL allow the inbound traffic on that port.

Any one missing produces the same symptom: a silent timeout with no obvious pointer to the actual cause. This is where most "why can't I reach my instance" tickets originate.

NAT Gateway: outbound only

A NAT gateway lives in a specific subnet in a specific AZ, performs source NAT for private instances, and has real hourly and per-GB cost. The packet walk for a private instance at 10.0.2.15 requesting a public registry:

Private subnet route table: 0.0.0.0/0 → nat-0abc...
NAT gateway rewrites source to its own Elastic IP + ephemeral port
NAT gateway's public subnet route table: 0.0.0.0/0 → IGW
IGW performs its own separate 1:1 NAT translation

Two distinct NAT translations — easy to collapse into one mental step, but they're separate resources doing separate jobs.

Why it lives in the public subnet: the NAT gateway needs its own route to the IGW, so it must sit in a subnet whose route table already points to the IGW. The private subnet's route table then points 0.0.0.0/0 at the NAT gateway. Two different route tables, two different subnets, one resource bridging them.

HA pattern: one NAT gateway per AZ, each AZ's private subnet routing to the NAT gateway in its own AZ. One NAT gateway for the whole VPC is cheaper but creates a single point of failure — if that AZ has an outage, every private subnet in every other AZ loses outbound internet access.

Cost trap worth auditing: traffic to S3 and DynamoDB from private subnets doesn't need to go through NAT at all if you use VPC Gateway Endpoints. Routing S3 traffic through NAT is billed per GB with no benefit over a free Gateway Endpoint.

Read the Full Article

The summary covers the core mental model. The full article goes deeper on:

The reverse-engineering formula for subnet sizing applied to autoscaling groups and EKS node groups, with the specific IP exhaustion failure mode explained
The full route table example with VPC peering and S3 Gateway Endpoint entries, and why most-specific-match matters for overlapping routes
IGW statelessness and why NACLs require explicit ephemeral port rules (1024–65535) that security groups handle automatically
NAT Gateway connection tracking limits: 55,000 concurrent connections per unique destination, PortAllocationErrors in CloudWatch as the signal, and when to reconsider architecture vs. adding more NAT gateways
The full security group vs. NACL comparison — stateful vs. stateless evaluation, allow-only vs. allow-and-deny, and why the default NACL and default security group behave differently out of the box

👉 AWS VPC Networking Fundamentals — Full Article

Microservices by Default: The Organizational Constraints Nobody Puts in the Architecture Diagram

Alok Ranjan Daftuar — Tue, 25 Aug 2026 06:29:39 +0000

"We're moving to microservices" has been the architectural ambition of engineering teams for a decade. More than 40% of organisations now report regretting at least some of those decisions. One team's consolidation back to a monolith: response times improved 13x, AWS costs dropped from $18K to $2.4K/month, deployment time fell from 45 minutes to 6 minutes.

These are not teams that failed at microservices. They implemented them correctly — and discovered the operational premium was not proportional to the benefit.

The Distributed System Tax

Every network boundary introduces failure modes that don't exist in-process: retries, circuit breakers, idempotency, distributed tracing, timeout budgets, schema versioning. This tax consumes 30–50% of engineering capacity once you cross the process boundary.

What you pay per service boundary:

Network:      Retries, circuit breakers, timeouts, idempotency keys
Observability: Distributed tracing, per-service dashboards, N on-call rotations
Deployment:   N pipelines, N rollback procedures, compatibility windows
Data:         No foreign keys, eventual consistency, data duplication, sagas

Multiply by number of services. Then ask: does the benefit justify this?

Conway's Law Is a Constraint, Not a Suggestion

Microservices deliver their promise only when the team structure supports end-to-end ownership without cross-team coordination for routine changes.

Developer:  "We're splitting into order-service, inventory-service,
             and payment-service."
Architect:  "Which team owns each?"
Developer:  "The same team. Us."
Architect:  "So you're paying the distributed system tax for three
             services, with no autonomy dividend, because one team
             owns all three."
Developer:  "..."
Architect:  "How many engineers do you have?"
Developer:  "Seven."

The Inverse Conway Maneuver: restructure teams first. The architecture follows. Decomposing architecture first and hoping teams catch up is how seven engineers end up debugging a distributed trace across five services for a bug that would have been a five-line stack trace in a monolith.

When Microservices Are Premature

Fewer than three independent teams — communication overhead of service boundaries exceeds coordination overhead of a shared codebase
Domains still evolving — wrong service boundaries require data migration and interface deprecation; wrong module boundaries are a refactoring task
Primary bottleneck isn't deployment frequency — microservices won't fix slow tests, hiring, or infrastructure cost

When Microservices ARE Worth It

Multiple autonomous teams with clear domain ownership
Genuinely divergent scaling requirements (>5x difference between components)
Compliance isolation requirements (PCI-DSS, HIPAA)
High deployment frequency at scale (dozens of deploys/day across teams)

The Modular Monolith — The Default That Gets Skipped

A single deployable with enforced module boundaries aligned to business domains. Not a monolith with good intentions — a monolith where boundaries are enforced by build tooling.

/src/modules
  /orders/api          ← public interface (what other modules may call)
  /orders/domain       ← internal (not accessible from other modules)
  /payments/api
  /payments/domain
  /inventory/api
  /inventory/domain

Rule: orders may only import from payments/api
      Enforced by ArchUnit, NDepend, or equivalent

Benefit: when you extract payments into a service,
         the boundary is already clean. The API is already defined.
         The migration is a deployment change, not a redesign.

The Decomposition Decision Framework

Step 1: What specific problem is decomposition solving?
Step 2: Do the organisational prerequisites exist?
        - Team exists to own the service end-to-end?
        - Team can deploy without coordinating with others?
        - Service interface is stable?
Step 3: Use the strangler fig — extract one bounded context at a time

Warning Signs

You have a distributed monolith:

Services deployed together in fixed order
Changes to Service A always require changes to Service B
Production incidents involve debugging 4+ services simultaneously

Your monolith doesn't need decomposing:

Deploy in under 10 minutes
Teams work in clearly separated namespaces
On-call incidents debuggable with a single log stream
Can scale horizontally without issues

Read the Full Article

This is a summary of the fourth post in the Cloud Defaults Reconsidered series. The full article includes the complete distributed system tax breakdown, Conway's Law analysis, modular monolith implementation patterns, and the full decomposition decision framework:

👉 Microservices by Default — Full Article

The full article includes:

Complete distributed system tax enumeration (network, observability, deployment, data)
Conway's Law and the Inverse Conway Maneuver in depth
Three common misconceptions debunked (scaling, deployment safety, "get boundaries right later")
Modular monolith implementation with enforcement tooling
Warning signs for distributed monolith vs well-structured monolith
Full decomposition decision framework (pain point identification, prerequisite checklist, strangler fig extraction)
Key takeaways with actionable thresholds

GitOps Repo Structure and Application Patterns: App-of-Apps, ApplicationSets, and AppProjects

Alok Ranjan Daftuar — Wed, 19 Aug 2026 09:34:10 +0000

Once you have ArgoCD running across clusters — hub-and-spoke or per-cluster — the next question is how to organize what it deploys. How should Git repositories be structured? When does App-of-Apps break down? What actually stops one team from deploying into another team's namespace?

Those decisions compound. Get the repo structure wrong and every later choice about ApplicationSets and AppProjects inherits the mess.

Repo Structure: Where Change Friction Lives

Three shapes dominate in practice. The choice isn't aesthetic — it determines where friction lives.

Mono-repo — one repository, all teams, all environments, organized by directory.

Wins: one place to search, one PR history, easy to see what's deployed where
Costs: PR review load and merge contention scale with team count; access control is directory-level CODEOWNERS, not repo permissions — weaker isolation than compliance-conscious teams want

Repo-per-team — each team owns a repository for everything it deploys.

Wins: access control is a repo permission; a team's deploy cadence doesn't create merge contention with others; blast radius of a bad commit is scoped to one team
Costs: cross-cutting changes (shared base image bump, cluster-wide policy) now touch N repos

Repo-per-app — finest grain, one repo per service.

Wins: tightest blast radius and access control
Costs: repo sprawl; cross-app changes are the worst version of the mono-repo problem — N repos, each requiring a separate PR

Service count	Team count	Access control need	Recommended structure
Under 8	1 platform team	Low	Mono-repo
8–30	2–5 independent teams	Moderate	Repo-per-team
30+	5+ teams, some with regulatory isolation	High	Repo-per-app, or repo-per-team with sensitive services split out

App-of-Apps: Simple, Until It Isn't

The simplest multi-Application pattern: a root Application whose only job is to deploy more Applications.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-checkout-team
  namespace: argocd
spec:
  project: checkout-team
  source:
    repoURL: https://github.com/example-org/checkout-team-gitops.git
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

It breaks down in two predictable ways at scale:

Sync-wave ordering is not a dependency gate. A shared-config Application at sync-wave: "0" and a checkout-service at sync-wave: "1" looks correct — until shared-config fails to sync. ArgoCD's wave ordering governs when a sync is attempted, not whether it's a hard prerequisite. checkout-service still syncs on schedule, its Deployment references a ConfigMap key that was never written, and the failure surfaces as a CrashLoopBackOff with no obvious link to the upstream cause.

Adding a new cluster or environment means hand-writing a new Application manifest every time. At 3 clusters and 5 services that's 15 manifests. At 12 clusters and 30 services it's 360 — every one written, not generated.

ApplicationSets: The Scaling Answer

Instead of writing N Application manifests, write one ApplicationSet with a generator that produces them.

The generators that matter:

Cluster — generates one Application per registered cluster, optionally filtered by label. Register a new spoke cluster and Applications for it appear automatically.
Git directory — generates Applications from repo structure: one per subdirectory. Add a new service directory and a new Application appears — no ApplicationSet edit required.
Matrix — combines two generators, producing the cross-product. Cluster × Git directory is the combination that matters most: every service in the repo, deployed to every cluster carrying a matching label.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: checkout-team-services
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - git:
              repoURL: https://github.com/example-org/checkout-team-gitops.git
              revision: main
              directories:
                - path: apps/*
          - clusters:
              selector:
                matchLabels:
                  team: checkout
                  env: prod
  template:
    metadata:
      name: '{{path.basename}}-{{name}}'
    spec:
      project: checkout-team
      source:
        repoURL: https://github.com/example-org/checkout-team-gitops.git
        targetRevision: main
        path: '{{path}}'
      destination:
        server: '{{server}}'
        namespace: checkout
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Add a new service directory — it deploys to every matching cluster automatically. Register a new cluster with the right labels — every existing service deploys to it automatically. Neither event requires touching the ApplicationSet.

That same automatic fan-out is why an ApplicationSet change deserves a dry run before it's applied — a mistyped label selector doesn't fail one Application, it fails every Application it would have generated:

argocd appset generate applicationset.yaml

This renders the full set of Applications the ApplicationSet would produce without applying anything — catching "this now matches four more clusters than intended" before it becomes a live sync.

AppProjects: The Multi-Tenancy Boundary

Everything so far describes how Applications get created. It says nothing about what stops one team's ApplicationSet from deploying into another team's namespace. That boundary is AppProject.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: checkout-team
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/example-org/checkout-team-gitops.git
  destinations:
    - server: https://spoke-eks-prod-us-east-1
      namespace: checkout
    - server: https://spoke-eks-prod-us-east-1
      namespace: cart
    - server: https://spoke-aks-prod-westeurope
      namespace: checkout
  clusterResourceWhitelist: []
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota
    - group: ""
      kind: LimitRange
  roles:
    - name: checkout-team-sync
      policies:
        - p, proj:checkout-team:checkout-team-sync, applications, sync, checkout-team/*, allow
      groups:
        - checkout-team-engineers

Three things worth being deliberate about:

sourceRepos should list the team's actual repos, not *. A wildcarded source list means any Application in this project can pull manifests from anywhere — defeating the point of repo-per-team access control.

destinations should be an explicit cluster/namespace list, not a wildcard. This is the line that stops a checkout-team ApplicationSet, misconfigured or compromised, from deploying into a payments-team namespace. A Matrix generator with a slightly-too-broad label selector (env: prod instead of team: checkout, env: prod) will happily generate Applications targeting every production cluster it can see. Without an AppProject destinations list, ArgoCD will attempt every one of them. The label selector mistake is the proximate cause; the AppProject destinations list is what turns that mistake into a rejected sync instead of an actual cross-team deployment.

clusterResourceWhitelist: [] is a deliberately restrictive default. Most application teams have no legitimate reason to create ClusterRoles or Namespaces through their own AppProject. Reserve that capability for a separate, tightly held platform-team project.

How the Three Pieces Compose

Repo structure determines what the ApplicationSet's Git generator sees. The ApplicationSet's Matrix generator combines that with cluster registration to produce Applications — if you haven't read Multi-Cluster ArgoCD Architecture, that's the foundation this builds on. The AppProject referenced by every one of those Applications keeps the resulting sync operations inside the boundary that repo was ever supposed to have.

Skip AppProjects and the first two pieces still function — they just function without a backstop, which tends to be fine until the day it very much isn't.

Read the Full Article

The full article covers:

Detailed cost/benefit analysis for all three repo structure shapes with the decision table
App-of-Apps sync-wave failure mode explained in full with the CrashLoopBackOff scenario
Complete Matrix generator YAML with all four ApplicationSet generator types explained
argocd appset generate dry-run workflow for catching label selector mistakes before they go live
Full AppProject YAML with sourceRepos, destinations, clusterResourceWhitelist, namespaceResourceBlacklist, and role definitions
The cross-team deployment scenario that AppProject destinations prevent — and why the label selector mistake alone isn't enough to cause it

👉 GitOps Repo Structure and Application Patterns — Full Article

Multi-Agent Systems Architecture: Patterns, Pitfalls, and Production Reality

Alok Ranjan Daftuar — Mon, 17 Aug 2026 06:32:23 +0000

An agent stuck in an infinite retry loop doesn't show up in your error rate. It shows up in your AWS bill — eleven days later.

A team ran a multi-agent system for eleven days. Latency normal. Error rate 0.0%. Every dashboard green. Cloud bill: $47,000. The agents were stuck in an infinite retry loop the entire time — because "wrong" doesn't show up in Grafana.

This incident is representative of a failure class unique to multi-agent architecture: agents operating correctly by every infrastructure metric while producing incorrect, looping, or redundant behavior at the application layer.

When Multi-Agent Is Not the Answer

Most teams reach for multi-agent too early. Stay with a single agent when:

The task fits in one context window (200K Claude / 1M Gemini)
Latency SLA is under 10 seconds
Your single-agent baseline isn't tuned yet
The work is sequential (no parallelism gain)
You don't have observability infrastructure

Multi-agent earns its complexity for: parallel subtasks, distinct tool access requirements, or failure isolation needs.

The Four Coordination Patterns

Orchestrator-Worker (~70% of production deployments): central planner decomposes tasks, dispatches to specialist workers, synthesizes results. Single point of failure but best observability.

Sequential Pipeline: fixed linear chain. Deterministic, easy to debug, but latency stacks.

Dynamic Handoff (Swarm): no central coordinator — agents decide who handles what at runtime. Flexible but prone to infinite handoff loops.

Hierarchical: orchestrators managing sub-orchestrators. For large-scale decomposition where a single orchestrator can't hold the full planning state.

State Persistence: Demo vs Production

The single capability that separates them: persisting intermediate results so workflows can pause, resume, and retry individual failed agents without restarting from scratch. Checkpoint at every task-level status transition to durable storage (Postgres, Temporal).

Inter-Agent Contracts

MCP for agent-to-tool communication. A2A for agent-to-agent. Beyond protocols: every handoff needs a typed, validated Pydantic model. Validation runs at the boundary — not three agents downstream when bad data causes inscrutable errors.

Cost Controls

Budget per workflow run — total token budget tracked and enforced across all agents
Model tier routing — frontier model only for reasoning-heavy steps; Haiku for routing/classification/formatting
Hard retry ceiling — attempt_count >= 3 terminates, not retries forever
Sequential before parallel — fan-out increases total spend even when it reduces latency

Observability: Detecting "Wrong"

Standard metrics detect whether agents are running. Multi-agent systems need metrics that detect whether agents are making progress:

Workflow completion rate — not error rate
Retry anomaly detection — total retries significantly exceeding task count = stuck loop
Cost per workflow run — token counts aggregated per workflow ID
Per-agent failure rate — which role is the common failure point

The $47K alert: flag any workflow where sum(attempt_count) > 2 * len(tasks).

Read the Full Article

This is a summary of my deep dive into multi-agent production architecture. The full article covers all patterns with implementation examples:

👉 Multi-Agent Systems Architecture — Full Article

The full article includes:

When multi-agent is NOT the answer (5 heuristics)
Four coordination patterns with architecture diagrams
Orchestration vs choreography comparison table
WorkflowState persistence with checkpoint-level granularity (Python dataclasses)
Inter-agent contracts with Pydantic validation
MCP and A2A protocol roles explained
Circuit breaker implementation for agent calls
Saga pattern for side-effectful agent workflows
Cost control patterns (budget caps, model tiers, retry ceilings)
OTel tracing decorator for agent calls
Retry anomaly detection alert logic
Production deployment checklist (15 items)

Service Mesh Everywhere? The Operational Cost of Cluster-Wide mTLS

Alok Ranjan Daftuar — Fri, 14 Aug 2026 07:05:14 +0000

"Add a service mesh" has become the Kubernetes equivalent of "make everything private." The reasoning seems unassailable: mTLS, observability, traffic management — all handled transparently. But the recommendation rarely comes with the operational bill attached.

At 1,000 pods, traditional Envoy sidecars consume approximately 70 GB of memory — before a single byte of application traffic. The latency overhead of Istio's traditional sidecar mTLS: +166% at high load. Compared to 8% for Istio Ambient and 33% for Linkerd.

What a Service Mesh Actually Provides

mTLS — encrypted, mutually authenticated pod-to-pod communication
Traffic management — weighted routing, canary, fault injection, circuit breaking
Observability — automatic golden signals for every service-to-service call
Policy — authorisation rules enforced at the network layer

What it does NOT provide: free performance, simpler operations, automatic security, or a substitute for application-level observability.

The Real Cost

Traditional sidecar overhead (Istio + Envoy):
  1,000 pods: ~72 GB memory overhead
  Monthly cost for proxy memory alone: ~$4,284

Latency at high load:
  Istio sidecar:  +166%
  Linkerd:        +33%
  Istio Ambient:  +8%
  Cilium (L7):    +99%

Plus: operational overhead of debugging Envoy config, pilot reconciliation, sidecar injection, and mesh upgrade coordination.

When a Mesh Is Unnecessary

< 10 services — NetworkPolicy + cert-manager covers it at a fraction of the cost
Dev/staging clusters — adds debugging surface without production benefit
Monolithic deployments — one service-to-service call doesn't justify a mesh
Teams without dedicated platform ownership — the mesh will add more incidents than it prevents

When a Mesh IS Worth It

50+ services across multiple teams with genuine policy requirements
Canary deployments and traffic shaping at scale
Regulated environments with explicit mTLS and auditability requirements

The 2026 Landscape Has Changed

Istio Ambient Mode (GA in 1.25): per-node ztunnel handles L4 mTLS, 90%+ memory reduction, 8% latency overhead.

Cilium Service Mesh: eBPF at kernel level, no sidecar, 40–60% network overhead reduction vs traditional proxies.

If overhead was your objection before mid-2025, re-evaluate.

Lighter Alternatives

mTLS: cert-manager + NetworkPolicy + IRSA/Workload Identity
Observability: OpenTelemetry with DaemonSet collector
Traffic management: Argo Rollouts + NGINX Ingress weighted routing

Read the Full Article

This is a summary of the third post in the Cloud Defaults Reconsidered series. The full article includes the complete decision framework, detailed cost breakdowns, operational overhead analysis, and architecture selection guide:

👉 Service Mesh Everywhere? — Full Article

The full article includes:

Memory and latency overhead calculations at scale
Operational debugging scenarios and upgrade coordination costs
Common misconceptions debunked (mTLS ≠ mesh, "free" observability, upgrade safety)
When a mesh is unnecessary vs when it's justified
Istio Ambient vs Cilium vs Linkerd vs traditional sidecars comparison
Complete 3-step decision framework
Lighter alternatives that close 80% of the gap

Multi-Cluster ArgoCD Architecture: Hub-and-Spoke vs. Per-Cluster, Done Right

Alok Ranjan Daftuar — Mon, 10 Aug 2026 05:48:33 +0000

Every ArgoCD tutorial ends with one cluster, one ArgoCD instance, and kubectl config current-context pointing at the same place ArgoCD is installed. That works fine — until you have two clusters. By the time you're at 10 or 15, across AWS and Azure, the architecture you picked on day one is either quietly paying for itself or quietly costing you an incident a quarter.

The two dominant patterns — hub-and-spoke and ArgoCD-per-cluster — are both reasonable in the right context and both wrong when applied past the cluster count and team topology they were designed for.

The Two Bad Defaults Teams Back Into

Rather than choosing deliberately, most teams end up at one of two failure modes:

ArgoCD-per-cluster sprawl — every cluster gets its own ArgoCD because that's what the getting-started guide showed. Nobody has a single view of what's deployed where. Upgrading ArgoCD becomes N separate change requests.
A single hub that becomes a bottleneck — one team stands up ArgoCD once, registers every cluster against it, and doesn't revisit that decision until the application controller is falling behind on reconciliation or a hub outage takes down deployments for every team at once.

Pattern 1: Hub-and-Spoke

One management cluster runs ArgoCD. Every other cluster is registered as a remote destination via a Secret containing that cluster's API server address and credentials.

What you get: single pane of glass, centralized RBAC and SSO, one audit trail.

What it costs you: the hub becomes a scaling bottleneck at high cluster/Application count; blast radius covers every spoke if the hub is compromised; network reachability to every spoke's API server is a real design requirement, not just an IAM policy.

Two blast-radius shapes worth planning for specifically:

Redis under memory pressure — ArgoCD's application controller caches live-vs-desired state in Redis across every Application it manages. An OOM or eviction storm on that single Redis instance stalls reconciliation for every cluster the hub manages simultaneously. From the outside it looks like "ArgoCD is stuck everywhere" — the root cause is capacity planning on a component most teams treat as an implementation detail.
A quietly broken peering path — a security group rule tightened on one spoke's cluster security group doesn't fail loudly. It shows up as that spoke going Unknown in the UI while everything else stays green, easy to dismiss as a blip until someone needs to ship a fix to that cluster and can't.

Network Reachability — The Part Tutorials Skip

On AWS: VPC peering or Transit Gateway gets the hub's control plane traffic to each spoke's private EKS API server endpoint. Private API server endpoints are the right default for spoke clusters, but every private-by-default decision adds a network path the hub now has to be deliberately connected to.

On Azure: VNet peering or Azure Private Link between the hub's VNet and each spoke AKS cluster's VNet. Same shape of problem — the hub's egress needs a routable, authorized path to a spoke's control plane.

Controller Sharding Past 10 Clusters

A hub managing 12 spoke clusters with ~50 Applications each (600 total) on a single unsharded controller will start lagging live cluster state by 3-4 minutes during normal operation, and considerably longer after a bulk change. The fix is multiple controller replicas with shard annotations on cluster secrets:

# argocd-application-controller StatefulSet
spec:
  replicas: 4

# Cluster Secret with shard annotation
metadata:
  annotations:
    argocd.argoproj.io/shard: "2"

With ARGOCD_CONTROLLER_REPLICAS=4 and shard annotations spread across 12 spoke secrets, each replica watches 3 clusters instead of all 12 — reconciliation lag drops back to single-digit seconds. This change isn't complete without also moving to a Redis HA (Sentinel-backed) deployment, since a single Redis becomes the new bottleneck once four controller replicas hit it concurrently.

Pattern 2: ArgoCD-per-Cluster

Each cluster runs its own ArgoCD instance and manages only itself.

What you get: fault isolation, no cross-cluster network dependency, clean blast radius.

What it costs you: N places to upgrade and patch; fragmented visibility; RBAC and project config duplicated N times with all the drift that implies.

Where this genuinely wins: regulated environments where a cluster's isolation boundary is a compliance requirement; air-gapped clusters with no viable network path back to a central hub; edge deployments where each site operates independently.

Cluster Registration — EKS

The clean way to register a spoke EKS cluster is IRSA or EKS access entries — not a static kubeconfig with a long-lived token. The cluster secret uses the aws eks get-token exec plugin:

apiVersion: v1
kind: Secret
metadata:
  name: spoke-eks-prod-us-east-1
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: spoke-eks-prod-us-east-1
  server: https://EXAMPLE1234567890.gr7.us-east-1.eks.amazonaws.com
  config: |
    {
      "execProviderConfig": {
        "command": "aws",
        "args": [
          "eks", "get-token",
          "--cluster-name", "spoke-eks-prod",
          "--region", "us-east-1",
          "--role-arn", "arn:aws:iam::111122223333:role/argocd-hub-spoke-access"
        ],
        "apiVersion": "client.authentication.k8s.io/v1beta1"
      },
      "tlsClientConfig": { "insecure": false, "caData": "<base64 cluster CA>" }
    }

The full chain: create the IAM role on the spoke account with a trust policy scoped to the hub's IRSA role → create an EKS access entry mapping that role to a Kubernetes group → bind that group to a scoped ClusterRole (not cluster-admin) → grant the hub's IRSA service account sts:AssumeRole → apply the Secret → verify with argocd cluster list. A Successful status confirms the exec plugin chain worked end to end.

Cluster Registration — AKS

On AKS, the equivalent is Azure AD Workload Identity with the kubelogin exec plugin:

apiVersion: v1
kind: Secret
metadata:
  name: spoke-aks-prod-westeurope
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: spoke-aks-prod-westeurope
  server: https://spoke-aks-prod-dns-a1b2c3d4.hcp.westeurope.azmk8s.io
  config: |
    {
      "execProviderConfig": {
        "command": "kubelogin",
        "args": [
          "get-token",
          "--login", "workloadidentity",
          "--server-id", "6dae42f8-4368-4678-94ff-3960e28e3630"
        ],
        "apiVersion": "client.authentication.k8s.io/v1beta1"
      },
      "tlsClientConfig": { "insecure": false, "caData": "<base64 cluster CA>" }
    }

The full chain: create an Azure AD application + federated identity credential binding the hub's ArgoCD service account to it → grant that application a scoped AKS RBAC role on the spoke cluster → label the hub's ArgoCD service account for workload identity → apply the Secret → verify with argocd cluster list. Auth failures here are almost always a --subject mismatch in the federated credential or the workload identity label missing from the pod spec.

Decision Framework

Cluster count	Compliance boundary	Team topology	Recommended pattern
2	None	Single platform team	Hub-and-spoke
3-5	None	Single or lightly federated	Hub-and-spoke with AppProjects
5-10	Some (staging vs. prod)	Multiple product teams	Hub-and-spoke, watch controller sharding
10-15	Regulatory/contractual isolation on specific clusters	Multiple teams, some regulated	Hybrid — hub for general fleet, per-cluster for isolated outliers
Any	Air-gapped / no viable hub network path	Any	ArgoCD-per-cluster

The hybrid row in practice: a platform team running 13 clusters — 10 standard clusters registering against the hub via IRSA/workload identity, plus 3 PCI-scoped clusters each running their own ArgoCD with no network path back to the hub, syncing from a separate access-restricted Git repository. The platform team accepts fragmented visibility for those 3 clusters in exchange for not having to argue, in every audit cycle, that the hub's blast radius doesn't touch the PCI boundary.

Read the Full Article

The full article includes:

Hub-and-spoke network reachability deep-dive for EKS (VPC peering, private API server endpoints, Transit Gateway) and AKS (VNet peering, Azure Private Link)
Complete step-by-step EKS auth chain: IAM role creation, EKS access entry, ClusterRoleBinding, hub-side IRSA policy
Complete step-by-step AKS auth chain: Azure AD app, federated credential, AKS RBAC role assignment, workload identity label
Controller sharding mechanics and Redis HA requirements for fleets past 10-15 clusters
The trust-boundary thinking connecting IRSA/Workload Identity to ArgoCD's cluster auth model
What's coming in Article 2: repo structure, App-of-Apps, ApplicationSets, and AppProjects

👉 Multi-Cluster ArgoCD Architecture — Full Article