DEV Community: Alok Ranjan Daftuar

Progressive Delivery with Argo Rollouts: Canary, Blue-Green, and AnalysisTemplates That Actually Gate

Alok Ranjan Daftuar — Mon, 05 Oct 2026 11:40:13 +0000

A standard Kubernetes Deployment rolling update has no concept of "check whether this is actually working before replacing more Pods." It replaces old Pods with new ones at the pace you configured, and if the new version is broken, it finds out the same way your users do — by serving it to more and more of them until someone notices.

Argo Rollouts replaces the Deployment resource with a Rollout — API-compatible in its Pod template, but with an explicit strategy for how traffic shifts and, critically, automated checks that can halt or reverse that shift before it reaches everyone. This post covers the two strategies that matter in practice and the AnalysisTemplate mechanism that makes the gating real rather than cosmetic.

Why a Deployment Isn't Enough

A Deployment's rolling update is governed by maxSurge and maxUnavailable — how many extra Pods can exist during the rollout, how many can be unavailable. Neither is a correctness check. A Deployment will happily roll a broken version to 100% of Pods because nothing in its model asks "is this new version actually healthy" — only "are enough Pods reporting ready," which a broken-but-still-passing-liveness-probe version satisfies just fine.

The two strategies have fundamentally different blast-radius profiles:

Canary                                    Blue-Green
  100% ─┐                                   100% ─┐ old (active)
        │                                         │
   10%  │ old          10% new                new │ 100% (preview,
        │              ▲                           │      no live traffic)
   50%  │ old      50% │ new                        │
        │              │ analysis gate               │ prePromotionAnalysis
    0%  │              │ new: 100%                    │ gate
        └──────────────┴──────────────►          cutover: 100% new,
        traffic shifts in steps,                  0% old, in one move
        gated between each step

Canary spreads risk across several small, gated exposures. Blue-green concentrates verification into one pre-cutover check, then moves all traffic at once.

Canary Strategy

Canary shifts traffic to the new version incrementally, in named steps, with a pause (manual or timed) between each. The key YAML shape:

strategy:
  canary:
    steps:
      - setWeight: 10
      - pause: { duration: 5m }
      - analysis:
          templates:
            - templateName: checkout-success-rate
          args:
            - name: service-name
              value: checkout-service-canary
      - setWeight: 50
      - pause: { duration: 5m }
      - analysis:
          templates:
            - templateName: checkout-success-rate
      - setWeight: 100

Each setWeight step shifts that percentage of traffic to the new ReplicaSet via whichever traffic-management integration is configured — an Ingress controller, a service mesh, or a load balancer controller. The analysis steps between weight increases are where this stops being "a slower rolling update" and becomes an actual gate: the rollout does not proceed to the next setWeight unless the referenced AnalysisTemplate reports success.

The traffic-router gap worth knowing: if the Ingress controller isn't one of Rollouts' supported traffic-router plugins, it silently falls back to a ReplicaSet-ratio approximation. A single canary Pod out of 10 total is not the same as a precise 10% of requests. Confirming which traffic-router plugin is actually active — not just assumed — is worth doing before trusting setWeight values as precise.

Blue-Green Strategy

Blue-green skips the incremental traffic shift and instead runs the full new version alongside the full old version, with a single cutover once the new version is verified:

strategy:
  blueGreen:
    activeService: checkout-service-active
    previewService: checkout-service-preview
    autoPromotionEnabled: false
    prePromotionAnalysis:
      templates:
        - templateName: checkout-smoke-test
      args:
        - name: service-name
          value: checkout-service-preview
    scaleDownDelaySeconds: 300

The new version scales up fully behind previewService — reachable for testing, not yet receiving production traffic. prePromotionAnalysis runs against the preview version before any cutover is possible. With autoPromotionEnabled: false, cutover requires an explicit kubectl argo rollouts promote even after analysis passes — a deliberate human checkpoint on top of the automated one.

scaleDownDelaySeconds keeps the old ReplicaSet running for 5 minutes after cutover so a rollback is a service-selector flip back, not a fresh Pod startup. Setting it too low defeats the rollback advantage — if Pods take 90 seconds to become traffic-ready and scaleDownDelaySeconds is 30, the old ReplicaSet is already gone by the time a problem surfaces.

AnalysisTemplates — What Actually Gates the Rollout

Both strategies are only as good as the analysis behind them. An AnalysisTemplate defines a metric query and a success condition; Rollouts polls it on an interval and compares the result against thresholds:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: checkout-success-rate
  namespace: checkout
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 1m
      count: 5
      successCondition: result[0] >= 0.95
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}", status!~"5.."}[1m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[1m]))

This queries Prometheus for the canary's success rate every minute, five times, and requires each sample to hold at or above 95%. Rollouts supports the same pattern against Datadog, New Relic, CloudWatch, and others — the shape of "query a metric, define a success condition, sample N times" is the transferable part.

The two failure modes worth planning for explicitly:

A successCondition that's too loose (result[0] >= 0.50, left over from testing) will pass a canary that's failing half its requests. The gate is present but not gating.
A count too low to be statistically meaningful on a low-traffic service — one or two failures in a handful of requests swing the ratio well outside a 95% threshold on pure noise, triggering rollbacks on releases that were actually fine.

Both point to the same discipline: AnalysisTemplate thresholds need to be tuned against the actual traffic volume and baseline error rate of the specific service, not copied from another service's template.

Where This Sits Against CI/CD Pipeline Stages

Kargo's Stage-level verification (or a CI-to-GitOps handoff) answers "should this artifact be allowed into this environment at all." Argo Rollouts answers a different question, one layer down: "given that this artifact is now allowed into this environment, how carefully should it be exposed to that environment's traffic."

The two compose rather than compete. A Kargo Stage promoting into prod can just as easily be updating a Rollout resource as a plain Deployment. Nothing about progressive delivery requires picking a different CI-to-GitOps mechanism.

Running both isn't redundant defense in depth for its own sake — each catches a class of failure the other structurally cannot. A service can pass every pre-promotion check and still fail an in-flight Rollouts analysis: a load-dependent memory leak, a connection pool exhausting under real concurrency, a downstream dependency behaving differently under production traffic patterns than under a synthetic staging test.

Decision Framework

Situation	Recommended strategy
Stateless service, traffic-splitting infrastructure already in place	Canary — gradual blast-radius limiting, most granular control
No traffic-splitting infrastructure available	Blue-green, or canary via ReplicaSet ratio approximation
Release needs real-traffic-pattern validation before any production exposure	Blue-green — preview service gives you that validation window
Workload has strict version-coupling to a shared schema	Neither — fix the deployment pattern first

Read the Full Article

The full post covers:

Complete annotated YAML for both canary and blue-green Rollout resources
The exact traffic-router gap that makes setWeight values imprecise without a supported plugin
Why scaleDownDelaySeconds sizing directly determines whether blue-green's rollback advantage is real or theoretical
How AnalysisTemplate count and successCondition interact with low-traffic services to produce false rollbacks
The precise relationship between Kargo/CI promotion gates and in-flight Rollouts analysis — why both are required and what each catches that the other cannot

👉 Progressive Delivery with Argo Rollouts — Full Article

The MCP Stateless Revolution

Alok Ranjan Daftuar — Thu, 01 Oct 2026 10:58:38 +0000

For two years, running an MCP server in production meant building infrastructure that had nothing to do with what the server was actually supposed to do. The protocol's session model forced sticky routing, shared session stores, and affinity cookies onto teams who otherwise had no reason to build stateful infrastructure at all. On July 28, 2026, the MCP specification removed the session entirely — and with it, the justification for that entire category of workaround.

What the old transport forced on you

An MCP initialize handshake returned an Mcp-Session-Id header, and every subsequent request in that conversation had to carry the same ID back to the same server instance. Keeping that session alive in production meant one of three things: sticky sessions at the load balancer, a shared session store like Redis so any instance could serve any request, or a client-side retry strategy that tolerated session loss and re-initialized when a pod restarted.

Each solution adds a moving part that fails in a specific, familiar way during a rolling deployment. A pod gets terminated, its in-memory session state goes with it, and every client pinned to that pod either gets a connection reset mid-conversation or silently starts talking to a session that no longer exists.

# Pre-2026-07-28: sticky routing required
apiVersion: v1
kind: Service
metadata:
  name: mcp-server
  annotations:
    alb.ingress.kubernetes.io/target-group-attributes: |
      stickiness.enabled=true,
      stickiness.type=lb_cookie,
      stickiness.lb_cookie.duration_seconds=3600

What SEP-2567 actually removes

The core change removes protocol-level sessions and the Mcp-Session-Id header from the Streamable HTTP transport entirely — not deprecates, removes. Every request is now self-describing: protocol version, client identity, and capabilities travel in _meta on that request rather than being negotiated once and remembered by a pinned server instance.

This does not mean MCP servers can no longer carry state across calls. If a server needs to remember something between tool calls, it mints an explicit handle and returns it from the first call. The client passes that handle back as an ordinary argument — the same way any REST API has always handled state. What disappears is the requirement that the same pod be the one to look it up.

Header-based routing: the other half of the change

Statelessness solves scaling, but creates a new problem: how does a gateway make intelligent routing decisions without parsing every JSON-RPC body? SEP-2243 answers this: Streamable HTTP requests must now carry Mcp-Method and Mcp-Name headers. A load balancer can now route, throttle, and meter MCP traffic on headers — without ever being MCP-aware at the body level.

# AWS ALB post-2026-07-28 — no stickiness, header-based routing
annotations:
  alb.ingress.kubernetes.io/conditions.mcp-tasks: |
    [{"field":"http-header","httpHeaderConfig":{"httpHeaderName":"Mcp-Method","values":["tasks/*"]}}]
  alb.ingress.kubernetes.io/actions.mcp-tasks: |
    {"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"mcp-tasks-pool","servicePort":8443}]}}

Long-running task calls route to a differently-sized backend pool. Ordinary tool calls go elsewhere. No custom MCP-parsing logic in the gateway.

Per-request auth: the security implication

Removing the session also removes the one place authentication used to be established once and implicitly trusted for the rest of the conversation. Every request now carries its own client identity in _meta, which means authorization is evaluated per request. A revoked credential takes effect on the very next call instead of only once the session naturally expires. The infrastructure implication: your gateway now performs an identity check on every request rather than once per conversation — real added latency on every call, worth budgeting for explicitly when sizing the gateway layer.

The migration window

The specification's deprecation policy gives a twelve-month compatibility runway for features being phased out. What is removed outright — with no grace period — is the session header and initialize handshake. The realistic migration path is running both transport paths side by side behind the same ingress until every client you support has moved. The sticky-routing configuration doesn't disappear the day you upgrade — it stays live serving the old path while new traffic routes to the stateless pool through header-based rules.

Read the Full Article

This summary covers the core protocol changes and their infrastructure implications. The full article includes:

Complete before/after Kubernetes Service and Ingress YAML for both AWS ALB and Azure Application Gateway
The ttlMs/cacheScope caching signals (SEP-2549) and what they mean for CDN/edge caching
W3C Trace Context propagation in _meta (SEP-414) and distributed tracing implications
How the removal of Last-Event-ID stream resumability simplifies liveness probe design
Detailed dual-path migration strategy to avoid mid-migration outages

👉 The MCP Stateless Revolution — Full Article

VPC Endpoints: Gateway vs Interface, and the AWS Traffic That Shouldn't Touch the Internet

Alok Ranjan Daftuar — Mon, 28 Sep 2026 05:31:14 +0000

Traffic from a private subnet to S3 routes through your NAT gateway by default — and gets billed per GB for the privilege — when a free alternative has existed for years. VPC endpoints give AWS services a private, in-VPC address path that never touches the internet. There are two fundamentally different mechanisms, and understanding how they differ changes how you decide which services actually need one.

Why private subnet traffic hits NAT at all

A private subnet instance calling s3.ap-south-1.amazonaws.com does exactly what it would do calling any internet host: routes to the NAT gateway, gets source-translated, exits through the IGW, and comes back the same way. Both the instance and the S3 bucket are inside AWS — the traffic never needed to leave AWS's network. VPC endpoints fix this by giving the traffic a direct in-VPC path.

Gateway endpoints: a route table entry, nothing more

Gateway endpoints exist for exactly two services: S3 and DynamoDB. They are not network devices — they are route table targets. AWS injects a managed prefix list route automatically; traffic matching S3's IP ranges routes to the endpoint instead of NAT.

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.ap-south-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]
}

Cost: free. No hourly charge, no per-GB processing charge. This makes the S3 Gateway endpoint close to a strict upgrade — there is essentially no reason not to add it to every VPC with private subnets that touch S3.

The one real limitation: Gateway endpoints are associated with specific route tables, not the whole VPC. A new private subnet with its own route table falls back to NAT silently unless you explicitly associate the endpoint with that table too. Worth adding to any subnet-creation checklist.

Interface endpoints: a real ENI, backed by PrivateLink

Every other AWS service that supports VPC endpoints uses an Interface endpoint — a real Elastic Network Interface provisioned in a subnet you specify, with a private IP from that subnet's range, backed by AWS PrivateLink.

resource "aws_vpc_endpoint" "secretsmanager" {
  vpc_id             = aws_vpc.main.id
  service_name       = "com.amazonaws.ap-south-1.secretsmanager"
  vpc_endpoint_type  = "Interface"
  subnet_ids         = [aws_subnet.private_a.id, aws_subnet.private_b.id]
  security_group_ids = [aws_security_group.vpce_sm.id]
  private_dns_enabled = true
}

Because it's a real ENI, it has its own security group — Interface endpoint access is controlled by security group rules, not route table association. private_dns_enabled = true makes the endpoint transparent to application code: AWS overrides the public DNS name to resolve to the endpoint's private IP inside the VPC, so no SDK or application configuration change is needed.

Cost: the opposite of Gateway endpoints. Interface endpoints bill hourly per AZ plus per-GB data processed. "Add Interface endpoints for everything" is not automatically correct — it requires actual usage data.

Which services actually justify an Interface endpoint

Start with services every private instance calls constantly regardless of application logic:

STS — IAM role assumption is called far more often than people expect under the hood
CloudWatch Logs — continuous log shipping from busy applications
ECR — image pulls from private subnets, especially during autoscaling cold starts or frequent CI/CD deployments

These three tend to justify their hourly cost quickly in any account with meaningful private-subnet compute. Lower-traffic services — Secrets Manager called once at startup, SNS for occasional notifications — are worth evaluating with real Cost Explorer data rather than provisioning preemptively.

aws ce get-cost-and-usage \
  --time-period Start=2026-07-01,End=2026-08-01 \
  --granularity MONTHLY \
  --filter file://nat-gateway-filter.json \
  --metrics "UsageQuantity" "BlendedCost"

Side-by-side comparison

	Gateway Endpoint	Interface Endpoint
Mechanism	Route table entry	ENI backed by PrivateLink
Services	S3, DynamoDB only	Most other AWS services
Cost	Free	Hourly per AZ + per-GB processed
Security control	Existing SG/NACL on the instance	Own security group on the endpoint ENI
DNS	No change needed	Private DNS override, if enabled
Multi-subnet	Associate route table per subnet	Deploy ENI per AZ needed

Failure modes worth knowing before you're debugging under pressure

Private DNS conflicts in Transit Gateway / peered VPC designs — if two connected VPCs both have Interface endpoints for the same service with private DNS enabled, DNS resolution can become ambiguous. An instance in VPC-A may resolve the service name to VPC-B's endpoint ENI. Test this explicitly in hub-and-spoke designs.
Security group too narrow — a security group that misses the calling subnet's CIDR produces a connection timeout that looks identical to a routing failure, but is actually a security group issue at the endpoint itself.
Endpoint deployed in wrong AZ — the endpoint still functions cross-AZ, but incurs cross-AZ data transfer charges on top of the endpoint's per-GB cost. Deploy into every AZ that has calling instances.
Custom DHCP option set or Route 53 Resolver rule taking precedence — if the VPC has custom DNS resolution configured, the endpoint's private hosted zone override can be superseded and traffic reverts to NAT despite correct endpoint configuration.

Read the Full Article

This summary covers the core mechanisms and decision framework. The full article includes:

Complete Terraform for both Gateway and Interface endpoints with security group scoping
The full AWS CLI route table inspection workflow
Detailed cost modelling for Interface endpoint break-even analysis
Security group and NACL implications in depth
The specific action to take this week if you haven't added the S3 Gateway endpoint yet

👉 VPC Endpoints: Gateway vs Interface — Full Article

Secrets Management in a GitOps World: Sealed Secrets vs. External Secrets Operator vs. Vault

Alok Ranjan Daftuar — Thu, 24 Sep 2026 09:03:22 +0000

GitOps means everything a cluster needs lives in Git. Secrets very much don't belong in Git — not even in a private repo, because Git history is forever and "who had read access eighteen months ago" is a much harder question than "who has read access today."

Three approaches resolve that contradiction. Each breaks the "it's all just committed to Git" assumption in a different place, and each solves a different subset of the actual problem.

What "Solving" Secrets in GitOps Actually Means

Before comparing tools, it's worth being precise about what's being solved — because the three approaches solve different parts of it:

Never having plaintext in Git — not even encrypted-and-committed, if the encryption key itself can leak
Rotating secrets without a Git commit — if rotation requires a PR, it happens less often than it should
Scoping which cluster or namespace can see which secret — the same multi-tenancy question AppProjects answer for Applications, but for secret values

No single approach scores well on all three by default. Each is a different set of tradeoffs.

Sealed Secrets

Bitnami's Sealed Secrets takes the most literal path: encrypt the secret client-side with a public key, commit the resulting SealedSecret resource, and let an in-cluster controller — the only holder of the matching private key — decrypt it at apply time.

kubectl create secret generic checkout-db-creds \
  --dry-run=client \
  --from-literal=password='REPLACE_ME' \
  -o yaml | kubeseal --format yaml > checkout-db-creds-sealed.yaml

# safe to commit — only the in-cluster controller can decrypt this
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: checkout-db-creds
  namespace: checkout
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEQ...

What it gets you: nothing sensitive ever touches Git. No external dependency — self-contained controller, no Vault cluster or cloud secrets service required.

What it doesn't get you: rotation still requires a commit. Changing a database password means re-sealing and re-committing — same PR friction as any other manifest change. At 40 services rotating every 90 days, that's 40 separate re-seal-and-commit operations per quarter, each needing someone with kubeseal access to the right cluster's public key at the time rotation is due.

It's also cluster-scoped by design: a SealedSecret sealed for cluster A's public key is meaningless on cluster B. At multi-cluster scale, that means either re-sealing per cluster or restricting Sealed Secrets to genuinely cluster-local secrets.

External Secrets Operator

ESO inverts the model entirely: the secret's actual value never enters Git at all. What's committed is a reference — "fetch this key from AWS Secrets Manager" — and a controller resolves that reference into a real Secret by calling the secrets backend at sync time.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: checkout-db-creds
  namespace: checkout
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: checkout-db-creds
  data:
    - secretKey: password
      remoteRef:
        key: checkout-service/prod/db-password

This manifest is entirely safe to commit — there's no ciphertext, no encoded value. The sensitive part is the value at checkout-service/prod/db-password, which lives only in AWS Secrets Manager.

What it gets you: rotation with zero Git commits — update the value in Secrets Manager and every ExternalSecret referencing it picks up the change on its next refreshInterval cycle. Centralized secret storage that extends your existing cloud secrets service into Kubernetes rather than maintaining a separate story.

What it doesn't get you: the secrets backend must be reachable from every cluster running ESO — reintroducing the network-reachability planning from multi-cluster architecture, this time for every spoke's path to Secrets Manager or Key Vault.

One scoping risk worth calling out: a SecretStore backed by an overly broad IAM role — one that can read every key under checkout-service/* rather than the specific key a given ExternalSecret needs — means a misconfigured ExternalSecret in one namespace can pull secrets it was never meant to see. Scope the IAM policy behind each SecretStore to the narrowest key path prefix that namespace actually owns.

Vault Agent Injector

Where ESO pulls a secret into a Kubernetes Secret object, Vault's Kubernetes integrations more often avoid creating a Secret object at all. The Vault Agent Injector mounts secrets directly into a Pod's filesystem as files, at Pod startup, authenticated via Kubernetes' own service account token exchanged against Vault's Kubernetes auth method.

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: checkout-service
        vault.hashicorp.com/agent-inject-secret-db-creds: secret/data/checkout-service/prod/db
    spec:
      serviceAccountName: checkout-service
      containers:
        - name: checkout-service
          # application reads from /vault/secrets/db-creds — never from an env var

What it gets you: the closest thing to "the secret was never a Kubernetes API object" — no Secret resource exists to be over-broadly RBAC'd, listed via kubectl get secrets, or accidentally dumped in a debug script.

More importantly: Vault's database secrets engine can generate a unique, short-lived database username and password per Pod, on demand, with a lease duration Vault itself enforces. A 1-hour lease means Vault revokes that specific credential at the database level after an hour, independent of whether anyone remembered to rotate anything. This changes rotation from "a scheduled human task" into "an enforced property of every credential issued" — a meaningfully different compliance posture.

What it doesn't get you: simplicity. Running Vault well — unsealing, storage backend, HA, policy management — is a larger operational commitment than either of the other two approaches. If Vault isn't already part of your platform, adopting it purely to solve the GitOps secrets problem is usually the wrong-sized tool.

Decision Framework

Situation	Recommended approach
Small team, few clusters, low rotation cadence	Sealed Secrets — simplest, no external dependency
Already using AWS Secrets Manager or Azure Key Vault	External Secrets Operator — extends existing storage into Kubernetes
Multi-cluster fleet, secrets needed consistently across clusters	External Secrets Operator — reference model isn't cluster-scoped
Already running Vault for dynamic credentials or PKI	Vault Agent Injector — extends a tool you're already operating
Secrets where even a `Secret` object existing in the cluster is too much exposure	Vault injector — no `Secret` object is ever created

These aren't exclusive. Running ESO for the bulk of a fleet's secrets and reserving Vault's injector for the small set of credentials needing dynamic short-lived leases is a reasonable production posture — matching each secret's sensitivity to the approach that handles it well.

Read the Full Article

The full article covers:

The precise three-part definition of what "solving" secrets in GitOps actually means — and why no single tool covers all three
The 90-day rotation cost at 40 services with Sealed Secrets, and why it surfaces at compliance audit time
The cluster-scoping limitation of Sealed Secrets at multi-cluster scale
Full SecretStore + ExternalSecret YAML with IRSA/workload-identity auth chain
The IAM scoping risk with ESO and how it mirrors AppProject destinations restriction
Vault's dynamic credential model explained — why it changes rotation from a task to an enforced property
How all three approaches compose in a real fleet rather than requiring a single tool choice

👉 Secrets Management in a GitOps World — Full Article

Migrating Ubuntu Servers from Azure to AWS: The linux-azure Kernel Trap

Alok Ranjan Daftuar — Mon, 21 Sep 2026 07:34:51 +0000

You replicate a Ubuntu 24.04 server from Azure to AWS using MGN. The replication completes cleanly. The test launch succeeds. EC2 status checks pass. And then you try to SSH in and get nothing — no response, no timeout, just silence.

This is a specific failure mode that only affects Ubuntu servers that started life on Azure, and it's deceptive precisely because every layer of the migration tooling reports success. The problem is three layers down, in the kernel package itself.

Why Azure Ubuntu Images Are Different

Ubuntu's official Azure images ship with linux-azure — a kernel package built and tuned specifically for Hyper-V. It includes Hyper-V-specific drivers and, critically, does not include AWS's ena (Elastic Network Adapter) driver.

MGN replicates the disk exactly as it is, including that kernel. When the Nitro hypervisor presents network hardware to the guest OS, the running kernel has no ena module to handle it. The instance comes up with no functioning network interface — not because of security groups or routing, but because the OS literally cannot talk to the hardware AWS gave it.

EC2 status checks pass because they verify the hypervisor can reach the instance at a basic level, not that the guest OS has a working network stack. "Instance healthy" and "instance reachable" are not the same claim.

Confirming the Cause via EC2 Serial Console

Since SSH is unavailable, diagnosis happens through the EC2 Serial Console. Three commands confirm the root cause:

uname -r
# 6.8.0-1015-azure    <- the tell: "-azure" suffix on an AWS instance

dpkg -l | grep linux-image
# linux-image-6.8.0-1015-azure   installed
# (no linux-image-generic or linux-image-aws present)

lsmod | grep ena
# (no output — module not loaded)

The -azure suffix on uname -r while running as an EC2 instance is close to a definitive signal on its own. The absent ena entry in lsmod confirms the mechanism.

The Fix: Kernel Swap Before Cutover

Install linux-generic — Ubuntu's broad-coverage kernel that includes ena support — alongside the existing kernel, update GRUB, and only then remove the Azure-specific one:

sudo apt update
sudo apt install -y linux-generic linux-headers-generic

# Confirm GRUB will boot the generic kernel first
grep GRUB_DEFAULT /etc/default/grub
sudo update-grub

The removal step comes after confirming the generic kernel boots successfully:

# Only after a successful boot on linux-generic:
sudo apt remove --purge linux-azure linux-image-*-azure linux-headers-*-azure
sudo update-grub

Removing the old kernel before confirming the new one boots is how a routine kernel swap turns into a boot-loop with no fallback entry in the GRUB menu.

The cleanest sequencing is doing this on the source Azure VM before MGN replication begins — so the replicated disk already has the correct kernel. If the source can't be touched, do it on a test-launched instance and validate before cutover.

Baking It Into a Pre-Flight Check

For a migration wave with multiple servers, this is worth scripting into a pre-migration audit:

#!/bin/bash
KERNEL=$(uname -r)
if [[ "$KERNEL" == *azure* ]]; then
  echo "WARNING: $(hostname) is running $KERNEL — Azure-specific kernel detected."
  echo "This server needs linux-generic installed and GRUB updated before MGN cutover."
  exit 1
else
  echo "OK: $(hostname) running $KERNEL — no Azure-specific kernel detected."
  exit 0
fi

Running this across every source server during wave planning — before agent installation — turns a per-incident debugging exercise into a one-line pre-flight check.

The Second Layer: Network Interface Naming

Even after ena loads correctly, the network interface name the OS expects can be wrong. Azure and AWS use different predictable naming schemes, and the netplan config on the source server was written against Azure's naming. After the kernel swap, the ena-backed interface may come up under a different name than the config expects.

The robust fix is matching by driver rather than hardcoded name:

network:
  version: 2
  ethernets:
    ena-primary:
      match:
        driver: ena
      dhcp4: true

This survives future re-platforming without another manual edit.

Other Azure Dependencies to Audit in the Same Pass

While fixing the kernel, check for:

walinuxagent — the Azure Linux Agent retries calls to 169.254.169.254, the same link-local address AWS's instance metadata service uses. This collision can interfere with cloud-init and AWS tooling.
Azure-specific udev rules — stale rules referencing Azure device paths, usually harmless but worth clearing.
cloud-init datasource — if pinned to Azure in /etc/cloud/cloud.cfg.d/, it will hang or fail on AWS trying to find a datasource that doesn't exist.

Don't Stay on linux-generic — Move to linux-aws

linux-generic is the right kernel to get unblocked during migration. It's not the right long-term kernel for a production AWS instance. Ubuntu publishes linux-aws, tuned specifically for Nitro, with platform-specific patches and its own update cadence separate from the generic track.

sudo apt update
sudo apt install -y linux-aws linux-headers-aws
sudo update-grub

Same discipline: confirm it boots and networks correctly before removing linux-generic.

Read the Full Article

This summary covers the core diagnosis and fix. The full article includes:

Why EC2 status checks pass even when the network is completely broken
The exact GRUB default entry approach vs. GRUB_DEFAULT=0 trade-offs
The boot-loop scenario that results from removing the old kernel too early
Validating through MGN test launch vs. serial console — why they're not equivalent
The full audit checklist for Azure-sourced servers entering a migration wave

👉 Migrating Ubuntu 24.04 from Azure to AWS: The linux-azure Kernel Trap — Full Article

Why Agent Infrastructure Is Its Own Discipline

Alok Ranjan Daftuar — Fri, 18 Sep 2026 05:39:20 +0000

Ask a platform team how they're going to run their first production agent and you'll get a confident answer within ten seconds: containerize it, put it behind an ingress, wire up a Horizontal Pod Autoscaler, done. It's the same playbook that's shipped every stateless service for the last decade, and there's no obvious reason an agent should be different. It accepts a request. It returns a response. It's just a container.

That answer is wrong — and it's wrong in a way that doesn't show up in a demo. It shows up three weeks into production, when a pod gets killed mid-reasoning because a liveness probe decided a 40-second tool call was a hang, or when the autoscaler adds five replicas because CPU spiked on a single request calling six tools in sequence, or when two "sessions" turn out to share a K8s Service without anyone having reasoned about what that means for isolation.

The shape of an agent request vs. a microservice request

A typical microservice request has a shape platform engineers have spent fifteen years optimizing around: bounded latency, a single unit of compute per request, statelessness between requests, and a clear success/failure signal at the HTTP layer. An agent request breaks all four at once.

Standard microservice request:
  client -> service -> [DB/cache lookup] -> response
  Duration: milliseconds to low seconds
  Compute: roughly constant per request
  State: none carried between requests
  Outcome signal: HTTP status code

Agent request:
  client -> agent -> [reason] -> tool call 1 -> [reason] -> tool call 2
        -> [reason] -> tool call N -> [reason] -> response
  Duration: seconds to minutes, highly variable
  Compute: proportional to reasoning depth and tool fan-out
  State: conversation/task context carried across the entire chain
  Outcome signal: HTTP 200 with a semantically wrong answer is common

That last line is the one platform teams underestimate most. An agent that loops on a tool, calls the wrong one, or returns a plausible-but-incorrect result will still return a healthy status code. The infrastructure layer has no way to distinguish a correct run from a confidently wrong one.

Where the standard playbook breaks

Microservice assumption	Agent reality	Practical implication
Liveness = "process is alive and responsive"	A live agent can be legitimately unresponsive for 30–90 seconds mid-tool-call	Naive liveness probes kill healthy pods mid-reasoning
CPU/memory tracks load	Load tracks reasoning depth and tool fan-out, not CPU	HPA on CPU/memory over- or under-scales unpredictably
Requests are independent	A single task often spans multiple round trips carrying shared context	Session/state handling needs an explicit design, not an assumption
One replica serves any request	Tool credentials and context may be tenant-specific	Routing and isolation boundaries need to be architectural, not incidental
Timeout = failure	Timeout at 30s might just mean the agent is still reasoning	Timeout budgets have to be set per tool-call chain, not per request

The liveness probe problem, concretely

With periodSeconds: 10 and failureThreshold: 3, a standard probe kills a pod if /health doesn't respond within roughly 30 seconds — which is an entirely normal duration for a reasoning step that includes a tool call to a slow downstream API.

The fix isn't a bigger failureThreshold. It's separating "the process is alive" from "the process is making forward progress":

livenessProbe:
  httpGet:
    path: /health          # answers: is the process itself alive?
    port: 8080
  periodSeconds: 15
  timeoutSeconds: 5
  failureThreshold: 3      # ~45s of true unresponsiveness before restart

readinessProbe:
  httpGet:
    path: /ready            # answers: can this pod accept new work right now?
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

/health should do nothing more than confirm the process's event loop is running — never block on the status of an in-progress tool call. /ready reflects capacity: a pod mid-reasoning can report itself not-ready for new work without being treated as dead. For a request/response service these two questions have the same answer. For an agent, they routinely don't.

What changed in the protocol layer

A large part of the awkwardness in running MCP-based agents on Kubernetes came from the original protocol design, which required persistent, pinned sessions between a client and a specific server instance — the opposite of what horizontally scaled infrastructure wants. That constraint forced teams into sticky routing and shared session stores just to keep a conversation coherent across requests.

The July 2026 MCP specification revision removed the protocol-level session entirely. Any request can now land on any server instance, and applications that need to carry state across calls do it the way HTTP APIs always have — by minting an explicit handle passed back as an ordinary argument, rather than relying on the transport to remember. That single change removes an entire category of infrastructure workaround (sticky routing, pinned sessions, shared session stores) that used to be treated as unavoidable.

The second shift is A2A (Agent-to-Agent protocol), which reached v1.0 in early 2026. Where MCP governs how an agent talks to tools and data sources, A2A governs how agents talk to each other — and that distinction matters for infrastructure design. A multi-agent system where agents call each other over A2A has different routing, identity, and isolation requirements than one where a single orchestrator calls tools over MCP. Both patterns are in production today.

Why the cost of getting this wrong compounds quietly

A liveness probe that kills healthy pods doesn't fail loudly — it shows up as an elevated error rate attributed to "model flakiness" or "the tool API being unreliable," and teams spend weeks tuning retry logic against a problem that's actually a probe misconfiguration one layer down. An autoscaler tuned on the wrong signal doesn't fail either — it just runs 30% more replicas than the workload needs, indefinitely, because nobody has a reason to suspect the scaling metric itself.

Getting the infrastructure layer right doesn't guarantee correct agent behavior, but getting it wrong guarantees you can't tell the difference between an agent that's actually failing and one that's simply being run on infrastructure that wasn't built for it.

Read the Full Article

The summary covers the core argument. The full article goes deeper on:

The full breakdown of each assumption failure — autoscaling signal, session and state design, isolation boundaries, and timeout budget — with the specific production failure mode each one produces
Why EKS and AKS don't solve the shape problem by default, and how the implementations diverge (IRSA vs. workload identity federation, ALB Ingress vs. Application Gateway) even when the underlying design goal is identical
The complete naive vs. corrected deployment YAML side-by-side, with the exact reasoning behind each change

👉 Why Agent Infrastructure Is Its Own Discipline — Full Article

VPC Peering vs Transit Gateway: Choosing the Right AWS Inter-VPC Connectivity Model

Alok Ranjan Daftuar — Tue, 15 Sep 2026 11:35:40 +0000

The default assumption most teams start with is that VPC peering is the "simple" option and Transit Gateway is the "enterprise" option you graduate into once you're big enough. That framing is wrong often enough to cause real architectural pain. The actual decision isn't about company size — it's about topology shape.

Before comparing features, answer one question: does your connectivity requirement look like a mesh or a hub-and-spoke? That single test resolves most of the debate.

The math that kills peering at scale

A peering connection is non-transitive — A cannot reach C through B even if both connections exist. For N VPCs in a full mesh, the connections required are N * (N - 1) / 2:

VPCs	Peering connections	TGW attachments
3	3	3
5	10	5
10	45	10
20	190	20

At 10 VPCs you're managing 45 separate connections, each with its own route table entries on both sides. Transit Gateway needs exactly N attachments regardless of how many VPCs you have.

Setting up VPC Peering in Terraform

resource "aws_vpc_peering_connection" "app_to_shared" {
  vpc_id      = aws_vpc.app.id
  peer_vpc_id = aws_vpc.shared_services.id
  auto_accept = true
}

resource "aws_route" "app_to_shared" {
  route_table_id            = aws_route_table.app_private.id
  destination_cidr_block    = aws_vpc.shared_services.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.app_to_shared.id
}

resource "aws_route" "shared_to_app" {
  route_table_id            = aws_route_table.shared_private.id
  destination_cidr_block    = aws_vpc.app.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.app_to_shared.id
}

Both directions need an explicit route — peering doesn't propagate routes automatically. Every new VPC added to a peered mesh means touching route tables on every existing member.

Setting up Transit Gateway with segmented routing

resource "aws_ec2_transit_gateway" "main" {
  description                     = "tgw-central-networking"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"
}

resource "aws_ec2_transit_gateway_vpc_attachment" "app" {
  subnet_ids         = [aws_subnet.app_private_a.id, aws_subnet.app_private_b.id]
  transit_gateway_id = aws_ec2_transit_gateway.main.id
  vpc_id             = aws_vpc.app.id
}

The disable pair on both defaults is deliberate — it enables route table segmentation instead of every attachment automatically seeing every other attachment.

Route table segmentation: the feature peering can't replicate

A single Transit Gateway can have multiple route tables. Each attachment associates with one, giving you actual network segmentation without touching security groups or NACLs:

TGW Route Table: "production"
├── app-vpc attachment    (associated)
├── shared-vpc attachment (associated, propagated)
└── sandbox-vpc attachment (NOT associated — isolated by design)

TGW Route Table: "sandbox"
├── sandbox-vpc attachment (associated)
└── shared-vpc attachment  (associated, propagated)

This is the pattern most multi-account AWS Organizations setups converge on: one Transit Gateway per region, a small number of route tables representing trust boundaries, and VPCs associated into the appropriate table.

Cross-account via AWS RAM

Transit Gateway solves the multi-account case cleanly through Resource Access Manager — one account owns the TGW, shares it via RAM, and spoke accounts create their own attachments:

resource "aws_ram_resource_share" "tgw_share" {
  name                      = "tgw-network-share"
  allow_external_principals = false
}

resource "aws_ram_resource_association" "tgw" {
  resource_arn       = aws_ec2_transit_gateway.main.arn
  resource_share_arn = aws_ram_resource_share.tgw_share.arn
}

resource "aws_ram_principal_association" "spoke_account" {
  principal          = "222233334444"
  resource_share_arn = aws_ram_resource_share.tgw_share.arn
}

Peering works cross-account too, but the N(N-1)/2 problem gets worse because you're also managing acceptance workflows and cross-account IAM permissions for every connection.

DNS and security group gotchas

Routing connectivity doesn't automatically mean DNS works across it. For peering, DNS resolution support must be enabled explicitly on both sides — without it, private hosted zone records resolve to public IPs or fail entirely, which looks identical to a routing problem.

Security group referencing across peering connections is also limited — most cross-account or cross-region setups fall back to CIDR-based rules rather than SG-to-SG references. Transit Gateway doesn't change this constraint.

Troubleshooting in priority order

When "VPC-A can't reach VPC-B" comes in:

Route table on both sides — both directions need explicit routes
Connection/attachment state — pending-acceptance on peering, non-available TGW attachment
Security group and NACL on both ends
DNS resolution settings — easy to mistake for a routing failure
TGW route table association — an attachment not associated with the expected route table silently fails

Read the Full Article

The full post covers everything above in depth, plus:

The DNS resolution strategy for hub-and-spoke setups using Route 53 Resolver endpoints — why it needs to be planned alongside routing, not retrofitted
The full side-by-side comparison table across all dimensions (routing model, cost, cross-region, segmentation, best fit)
The migration path from peering to Transit Gateway — running both in parallel, using most-specific-match routing for a controlled cutover without an all-or-nothing switch
When peering is genuinely the right answer and why you shouldn't add TGW complexity to a two-VPC problem

👉 VPC Peering vs Transit Gateway — Full Article

CI to GitOps Handoff: Argo Image Updater vs. CI-Writes-the-Commit vs. Kargo

Alok Ranjan Daftuar — Thu, 10 Sep 2026 09:23:01 +0000

ArgoCD only reconciles what's in Git. A CI pipeline builds an image, pushes it to a registry — and then what? Something has to turn "a new image exists" into "a commit exists in the GitOps repo referencing it." That handoff is where a surprising number of GitOps implementations quietly go wrong: either it's done manually, or it's automated in a way that reintroduces the exact imperative-deploy risk GitOps was supposed to remove.

Three approaches cover almost every real setup. The choice isn't just operational preference — it interacts directly with your repo structure and how much promotion-stage enforcement you actually need.

The structural difference in one picture

Approach 1 — Argo Image Updater
  CI: build → push image ──────────────► registry
                                              │ (polled every 1-2 min)
                                              ▼
                                    Image Updater controller
                                              │ writes tag
                                              ▼
                                     GitOps repo commit ──► ArgoCD syncs

Approach 2 — CI writes the commit
  CI: build → push image ──► registry
       │
       └── same workflow, next job: bump tag, commit, push ──► GitOps repo ──► ArgoCD syncs

Approach 3 — Kargo
  CI: build → push image ──► registry
                                 │
                                 ▼
                            Kargo Warehouse (watches registry)
                                 │
                                 ▼
                          Stage: staging ──(verify)──► Stage: prod
                                 │                          │
                                 ▼                          ▼
                          GitOps repo commit          GitOps repo commit
                          (staging path)               (prod path, only if
                                                         staging verified)

Approaches 1 and 2 go straight from "new image" to "commit" with no concept of one environment gating another. Kargo inserts an explicit, enforced step between them. Everything below is detail on why you'd pick one over another.

Argo Image Updater

Image Updater runs alongside ArgoCD, polls registries for new tags matching a pattern, and writes the updated tag back into the GitOps repo. CI's job stays exactly "build, test, push image" — no GitOps-repo awareness required.

Configuration lives as annotations on the Application itself:

annotations:
  argocd-image-updater.argoproj.io/image-list: checkout=111122223333.dkr.ecr.us-east-1.amazonaws.com/checkout-service
  argocd-image-updater.argoproj.io/checkout.update-strategy: semver
  argocd-image-updater.argoproj.io/checkout.allow-tags: 'regexp:^v[0-9]+\.[0-9]+\.[0-9]+$'
  argocd-image-updater.argoproj.io/write-back-method: git
  argocd-image-updater.argoproj.io/git-branch: main

The write-back-method: git setting is the important one. The alternative (argocd) writes to the Application object only, meaning the running state and the Git repo can silently diverge — undermining the entire premise of GitOps as the source of truth. If you use Image Updater, git write-back is the only mode consistent with a real GitOps model.

Trade-off: polling-based (1-2 minute delay), and no native concept of promotion stages. Getting a build to flow dev → staging → prod with any gate means bolting that logic on separately — which is exactly the gap Kargo exists to close.

CI writes the manifest commit

No separate controller. The CI job itself pushes the commit after a successful build. The auth chain here deserves the same scrutiny as cloud credentials — a long-lived PAT with write access to the GitOps repo is the wrong answer for the same reasons a long-lived AWS access key is wrong.

The right pattern: a GitHub App scoped to exactly the one GitOps repo, with an installation token minted per workflow run:

jobs:
  update-gitops-repo:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Generate GitHub App token for GitOps repo
        id: app-token
        uses: actions/create-github-app-token@v1
        with:
          app-id: ${{ vars.GITOPS_BOT_APP_ID }}
          private-key: ${{ secrets.GITOPS_BOT_PRIVATE_KEY }}
          owner: example-org
          repositories: checkout-team-gitops
      - uses: actions/checkout@v6
        with:
          repository: example-org/checkout-team-gitops
          token: ${{ steps.app-token.outputs.token }}
      - name: Bump image tag
        run: yq -i '.image.tag = "${{ github.sha }}"' apps/checkout-service/prod/values.yaml
      - name: Commit and push
        run: |
          git config user.name "gitops-bot"
          git config user.email "gitops-bot@example.org"
          git commit -am "checkout-service: bump to ${{ github.sha }}"
          git push

The token is scoped to one installation (one repo) and expires roughly an hour after issuance — no long-lived credential sitting in a secrets store.

Trade-off: no delay, no extra controller. But CI now has push access to the thing that controls production. A GitHub App token scoped org-wide instead of to one repo turns a compromised CI secret from a one-service incident into an org-wide one. The scoping isn't optional hardening — it's the difference between those two outcomes.

Kargo

Image Updater and CI-writes-the-commit both update one Application at a time with no native concept of "this build must pass staging before prod." Kargo models that as a first-class API.

Two core resources: a Warehouse (watches for new artifacts, like Image Updater) and a Stage (one step in a promotion pipeline). The key field:

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: prod
  namespace: checkout-team
spec:
  requestedFreight:
    - origin:
        kind: Warehouse
        name: checkout-service
      sources:
        stages: [staging]   # prod can only promote freight that passed through staging

sources: stages: [staging] means prod is structurally incapable of promoting an image that hasn't gone through staging — enforced by Kargo, not by convention. Verification gates (smoke tests, analysis runs) attach to each Stage, and Kargo tracks exactly which artifact made it through which Stages.

The distinction from bolting a manual approval onto Image Updater: the gate is a property of the Stage graph itself, checked by Kargo before it constructs a promotion for the next Stage. Not a step someone has to remember to run.

Trade-off: another controller with its own resource model on top of ArgoCD's. Earns its cost specifically when multi-stage promotion with gated, auditable progression is a real requirement — not as a default for every service.

Decision framework

Situation	Recommended approach
Single environment, or environments promote independently	Argo Image Updater — simplest, no CI changes needed
Simple promotion logic already visible in CI (merge to main = staging, tag = prod)	CI writes the commit — keeps logic in one place, no extra controller
Hard requirement that prod only receives artifacts that passed staging with automated verification	Kargo — this is the exact problem it's built to solve
Regulated workloads needing an auditable promotion trail as a first-class object	Kargo — Stage/Freight model gives you that, not something reconstructed from CI logs

These aren't mutually exclusive across a platform — a team running mostly independent services on Image Updater might still put its most compliance-sensitive service through Kargo specifically.

Read the Full Article

The summary covers the shape of each approach and the key trade-offs. The full article goes deeper on:

The complete Image Updater Application manifest — image-list, update-strategy, allow-tags, and write-back-method in context, not just the annotation snippet
The full two-job GitHub Actions workflow (build + update-gitops-repo), including the step that verifies the App token fails against repos it shouldn't reach — confirming blast radius, not just that it works
The complete Kargo Warehouse + Stage YAML including the verification block that wires an AnalysisTemplate smoke test to the staging Stage — the part that makes "staging verified" mean something concrete
Why a Matrix generator with a too-broad cluster label selector connects directly to the auth chain here: AppProject destinations as the backstop when the ApplicationSet generates more than intended
Article 4 preview: secrets management (Sealed Secrets, External Secrets Operator, Vault plugin) — what each actually protects against and what it doesn't

👉 CI to GitOps Handoff: Argo Image Updater vs. CI-Writes-the-Commit vs. Kargo — Full Article

Always Encrypt in Transit: The Gap Between TLS Everywhere and Actual Transport Security

Alok Ranjan Daftuar — Tue, 08 Sep 2026 07:26:21 +0000

"We have TLS everywhere." It appears in every architecture review and every compliance questionnaire. The gap is not in the intention — it is in the implementation. Most systems that claim TLS everywhere have TLS at the edge and plaintext everywhere else: the ingress controller to the pod is HTTP, pod-to-pod traffic is unencrypted, and the application-to-database connection string never had Encrypt=True or sslmode=require set. None of these gaps appear in architecture diagrams.

This post maps where TLS is actually absent, why the gaps exist, and the implementation path that closes them — without defaulting to "add a service mesh" as the answer to every transport security question.

The Four Layers Where TLS Is Actually Absent

Before solutions, the precise inventory of where plaintext exists in systems claiming TLS everywhere:

Scenario: "Secure" architecture
  ✅ HTTPS on the load balancer
  ✅ Private subnets, no public IPs
  ✅ Security groups restricting access
  ❌ Ingress Controller → Pod: plaintext HTTP
  ❌ Pod → Pod: plaintext HTTP/gRPC
  ❌ Application → RDS: plaintext TCP
  ❌ Cluster infrastructure certs: expire annually, no automated rotation

What "TLS everywhere" actually means in this architecture:
  TLS on the edge. Plaintext the rest of the way.

Layer 1 — Ingress-to-pod: nginx-ingress and Traefik terminate TLS at the edge and forward plain HTTP to backend pods by default. The gap is invisible in architecture diagrams.

Layer 2 — Pod-to-pod: Kubernetes does not encrypt data plane traffic. A compromised pod on a shared node can observe plaintext traffic from other pods on the same node — the VPC boundary is not the relevant perimeter here, the pod boundary is.

Layer 3 — Application-to-database: Enabling TLS on RDS makes the database capable of TLS connections. It does not enforce them. A connection string without sslmode=require or Encrypt=True connects over plaintext regardless.

Layer 4 — Cluster infrastructure certs: kubeadm cluster CA certificates expire after one year by default with no automated renewal. A missed rotation takes down the entire cluster — not an application outage, a cluster-level failure where kubectl stops working entirely.

Termination Architecture: Where the Decision Gets Made

Where TLS terminates determines which traffic is encrypted. Three patterns:

Edge termination only — TLS terminates at the ingress controller, all internal traffic is plaintext. Legitimate for single-tenant clusters with no regulated data. Not legitimate to call "TLS everywhere."

Re-encryption — the ingress controller terminates external TLS and re-encrypts before forwarding to the pod. Closes the ingress-to-pod gap without a service mesh:

# NGINX Ingress: re-encrypt to backend over TLS
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-ingress
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
  - hosts: [api.yourdomain.com]
    secretName: api-tls-secret
  rules:
  - host: api.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 8443

End-to-end mTLS via service mesh — the mesh intercepts all pod-to-pod traffic and wraps it in mTLS. Strongest posture, highest operational overhead. Justified in regulated multi-tenant clusters; disproportionate for smaller service estates.

cert-manager: The Implementation That Actually Automates It

cert-manager handles issuance, renewal, and storage in Kubernetes Secrets with zero manual steps in the rotation path. One ClusterIssuer and one Certificate resource per service:

# ClusterIssuer: Let's Encrypt production
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@yourdomain.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx
---
# Certificate: 90-day validity, auto-renewed at 60 days remaining
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-certificate
  namespace: production
spec:
  secretName: api-tls-secret
  duration: 2160h
  renewBefore: 720h
  dnsNames:
  - api.yourdomain.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

For internal service-to-service certificates, use a self-signed CA issuer with 24-hour validity — cert-manager makes short-lived internal certs operationally trivial, and a 24-hour certificate has a dramatically smaller blast radius than a one-year certificate if the key is compromised:

# Internal service certificate — 24h validity, rotated automatically
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: order-service-cert
  namespace: production
spec:
  secretName: order-service-tls
  duration: 24h
  renewBefore: 8h
  dnsNames:
  - order-service.production.svc.cluster.local
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer

Certificate Lifecycle: Where Security Controls Become Outages

The failure mode is consistent: a certificate is issued, configured, and forgotten. The alert that was supposed to fire before expiry never got configured, or fired into a channel nobody watches. The certificate expires. The service goes down. The postmortem recommends "better monitoring" — until the same thing happens eighteen months later with a different certificate.

Alert on time-to-expiry, not on expiry. A Prometheus alert that fires at 15 days remaining gives the team time to investigate before the outage. An alert that fires when the certificate has already expired is a notification of an ongoing incident:

groups:
- name: certificate-expiry
  rules:
  - alert: CertificateExpiringIn15Days
    expr: |
      certmanager_certificate_expiration_timestamp_seconds
        - time() < (15 * 24 * 3600)
    for: 1h
    labels:
      severity: warning
    annotations:
      summary: "Certificate expiring soon"
  - alert: CertificateExpired
    expr: |
      certmanager_certificate_expiration_timestamp_seconds - time() < 0
    labels:
      severity: critical
    annotations:
      summary: "Certificate has EXPIRED"

Both alerts are required. Most teams have only the expired alert — which is a notification of an ongoing incident, not a prevention.

mTLS Without a Service Mesh

For fewer than twenty services, cert-manager internal CA plus application-level TLS provides mutual authentication without sidecar injection or a mesh control plane. The application handles the TLS handshake directly using certificates cert-manager issues and rotates automatically:

// .NET: present a client certificate and verify the server against the internal CA
var clientCert = X509Certificate2.CreateFromPemFile(clientCertPath, clientKeyPath);
var caCert     = new X509Certificate2(caCertPath);

var handler = new HttpClientHandler();
handler.ClientCertificates.Add(clientCert);
handler.ServerCertificateCustomValidationCallback = (_, cert, chain, _) =>
{
    chain!.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;
    chain.ChainPolicy.CustomTrustStore.Add(caCert);
    return chain.Build(cert!);
};

The certificates are mounted from Kubernetes Secrets that cert-manager manages. When cert-manager rotates the Secret, the volume mount is updated in place — no application restart required.

Read the Full Article

This summary covers the core gaps, termination architecture patterns, cert-manager setup, and the mTLS-without-mesh approach. The full article includes:

The complete database enforcement gap — rds.force_ssl, connection string Encrypt=True, and why enabling TLS on the DB is not the same as enforcing it
The VPC isolation misconception in full — why private subnets don't substitute for encryption
The "Kubernetes encrypts cluster traffic" misconception — what it actually encrypts vs. what it doesn't
Full decision framework mapping every traffic path to the right termination architecture
When regulated environments (PCI-DSS, HIPAA) make full end-to-end TLS non-negotiable
Eight key takeaways covering every layer of the in-transit security posture

👉 Always Encrypt in Transit: The Gap Between TLS Everywhere and Actual Transport Security — Full Article

AWS MGN Architecture: How Continuous Replication Actually Works

Alok Ranjan Daftuar — Thu, 03 Sep 2026 07:29:32 +0000

The first time I ran an AWS MGN migration, I trusted the console's green "Healthy" status without understanding what it actually meant. It took a stalled replication and a confusing lag metric mid-cutover to make me go back and learn the mechanics underneath the dashboard. This post is the breakdown I wish I'd had before that first wave.

MGN does one job: continuous, block-level replication of a source server's disks to a staging area in your target AWS account, so that when you cut over, you launch a fully synced, bootable EC2 instance instead of doing a one-time data copy and hoping nothing changed since. The distinction that matters is block-level, not file-level — MGN doesn't care about your filesystem or application state, which is what allows it to keep a target in near-real-time sync with a live, running source server throughout the migration.

The four components, in order

1. The replication agent — installed directly on the source server, hooks into the OS's disk I/O path at the kernel level. On Linux this means a kernel-level block device reader; on Windows, a filter driver. This low-level hook is why a kernel version mismatch on the source isn't a minor footnote — it's a direct threat to replication itself.

sudo python3 ./aws-replication-installer-init.py \
  --region ap-south-1 \
  --aws-access-key-id <access-key-id> \
  --aws-secret-access-key <secret-access-key> \
  --no-prompt

The --no-prompt flag matters for scripted wave installations — without it, the installer pauses for disk selection confirmation and silently stalls automation.

2. The staging area subnet — a designated subnet in the target AWS account where MGN provisions its replication infrastructure. Needs outbound connectivity and enough IOPS headroom on EBS to keep up with the source's write rate. Undersizing this subnet or placing it behind restrictive route tables is a common cause of replication that starts but never stabilizes.

3. The replication server and EBS volumes — for each source server, MGN provisions a temporary EC2 instance in the staging subnet whose only job is to receive the block-level stream and write it to EBS volumes that mirror the source's disk layout. These EBS volumes are the actual replica — they're what gets snapshotted at cutover.

The cost model people get wrong: you're paying for these replication servers and EBS volumes for the entire duration of the migration project, not just at cutover. A wave that sits in "replicating, not yet cut over" for six weeks accumulates staging infrastructure cost for all six weeks.

4. Continuous sync and the lag metric — once initial replication completes, MGN switches to continuous incremental sync: every write on the source is captured and streamed as it happens. The metric to watch is replication lag — how far behind the target EBS volumes are from the live source. Lag that climbs rather than staying flat is a leading indicator of a cutover that will either take much longer than expected or launch a target instance further behind the source than your rollback tolerance allows.

aws mgn describe-source-servers \
  --region ap-south-1 \
  --query 'items[*].{Server:sourceServerID, LagDuration:dataReplicationInfo.lagDuration, State:dataReplicationInfo.dataReplicationState}' \
  --output table

Use this as a scriptable pre-cutover gate rather than eyeballing a color indicator in the console.

What cutover actually triggers

Cutover is not a data operation — the data is already synced. What it actually does:

Takes a final snapshot of the target EBS volumes at the moment of cutover
Launches a new EC2 instance from that snapshot using the launch template you've configured
Runs the MGN post-launch conversion process — driver injection, network configuration, and OS-level changes needed to make a disk image that ran on a different hypervisor boot correctly under AWS's Nitro hypervisor

Step 3 is where kernel driver failures happen. The conversion process needs to inject or activate the correct network driver (ena) for the instance to have connectivity after launch. If the source server's kernel doesn't have the module available or the boot configuration doesn't reference it correctly, the instance can come up without network connectivity — or fail to boot entirely.

Launch settings most people under-invest in

Every source server has its own launch template in MGN. Worth setting deliberately before cutover, not left at defaults:

Target instance type — set explicitly for anything performance-sensitive rather than trusting MGN's right-size recommendation
Subnet and security groups — should be decided by your VPC CIDR plan, not chosen ad hoc at cutover time
IAM instance profile — doesn't carry over from the source server; must be set in the launch template explicitly
Test launch — MGN supports launching a test instance from current replication state without ending replication. Always run at least one test launch before the real cutover. Skipping this is the single most common way teams discover a launch-time problem during the actual cutover window instead of during a safe rehearsal

Replication settings that matter under load

Three settings worth understanding rather than leaving on defaults for source servers with high write throughput:

Bandwidth throttling — MGN doesn't cap replication bandwidth by default, which can compete with production traffic on a busy source server. Set a maximum throughput per source server for anything actively serving production load during the replication window
Staging volume type — the default EBS volume type for staging can itself become the bottleneck for high-IOPS sources, showing up as climbing lag even when network bandwidth is fine. Rule this in or out separately from the network-bandwidth cause when diagnosing a server that won't stabilize
Point-in-time (PIT) snapshots — MGN can retain a rolling window of snapshots of the staging volumes, giving you a recovery point earlier than "right now" if you need to launch from a known-good state. Configure this under Replication settings → Point-in-time snapshots in the MGN console before replication starts — snapshots only accumulate from the point the setting is enabled

Why continuous replication beats snapshot-and-copy

The snapshot approach has an unavoidable trade-off: the longer the gap between snapshot and cutover, the more source-side changes are missing from the target. Closing that gap means either accepting data loss or taking the source offline for the copy window.

MGN's continuous model removes that trade-off entirely. The target stays within seconds of the source right up until cutover, and the source never has to go offline for the migration itself — only for the brief cutover window when traffic is actually redirected. For anything with a real availability requirement, this is the entire reason to use MGN over a manual export/import process.

Read the Full Article

The summary covers the core mechanics. The full article goes deeper on:

The troubleshooting priority order for "these are definitely connected but nothing works" tickets — route tables, attachment state, security layers, DNS — and why checking in that sequence resolves the large majority of cases without reasoning about the full topology from scratch
The specific kernel driver failure mode that surfaces when migrating Azure-sourced Ubuntu images to AWS, why it manifests the way it does, and the fix
The full cost model for staging infrastructure across a multi-week wave, with the specific line items to flag before the wave kicks off
CIDR planning considerations for the staging subnet relative to the rest of the target VPC

👉 AWS MGN Architecture: How Continuous Replication Actually Works — Full Article

Enforcing Modular Monolith Boundaries in .NET: NDepend, Parallel Pipelines, and the Architecture That Holds

Alok Ranjan Daftuar — Mon, 31 Aug 2026 06:03:32 +0000

A modular monolith without enforcement is not an architecture — it is a monolith with good intentions.

The Problem

Most teams skip the modular monolith and jump straight to microservices. The ones that do attempt a modular monolith rely on convention — "don't cross module boundaries" — which fails the moment deadlines hit.

The difference between a well-structured modular monolith and a mess is whether boundaries are maintained by tooling or by convention.

The Solution Structure

Each module is a pair of .NET projects:

src/Modules/
  Orders/
    YourApp.Orders/              ← internal: domain, application, infrastructure
    YourApp.Orders.Contracts/    ← public: DTOs, interfaces, events
  Payments/
    YourApp.Payments/
    YourApp.Payments.Contracts/

The rule: modules may only reference each other's *.Contracts projects. The compiler enforces this physically — no project reference means no type access.

Four Layers of Enforcement

Compiler — project references prevent cross-module type access
NetArchTest — architecture tests fail the build on namespace-level violations
NDepend CQLinq — catches dependency cycles and coupling the compiler can't see
Quality Gates — block PRs that introduce new boundary violations

Module-Scoped Data

Each module owns a dedicated DbContext with a schema prefix (orders.*, payments.*). No module queries another module's tables.

Cross-Module Communication

Modules communicate via MediatR in-process events. Orders publishes OrderPlaced; Payments subscribes — without Orders knowing Payments exists.

This is also the extraction seam: when you eventually extract a module into a service, MediatR becomes a message broker. The event contract stays the same.

Parallel CI

strategy:
  matrix:
    module: [Orders, Payments, Inventory]
  fail-fast: false

Each module's tests run in parallel. CI time scales with the slowest module, not the total count.

The Extraction Path

When a module genuinely needs independence:

Add outbox table → publish to real broker
Replace MediatR handlers with broker consumers
Deploy module as separate service
Publish *.Contracts as NuGet package

The boundary was already clean. Extraction is a deployment change, not a redesign.

The full post covers NDepend CQLinq rule examples, Quality Gate configuration, GitHub Actions pipeline YAML, test isolation patterns, and a production checklist.

👉 Read the complete implementation guide

AWS VPC Networking Fundamentals: VPCs, Subnets, CIDR, Route Tables, IGW, and NAT Gateways

Alok Ranjan Daftuar — Fri, 28 Aug 2026 03:49:00 +0000

If you've provisioned a VPC from a Terraform module without fully internalising what each piece is doing, that's fine — right up until something breaks. An instance that should be reachable isn't. A private instance can't pull a package update. And you're left checking five different resources with no clear mental model of how they connect.

This post builds that mental model from the ground up. Not just definitions — the why behind each piece, so troubleshooting becomes deduction instead of guesswork.

CIDR math you actually need

A CIDR block is IP address / prefix length. The prefix length fixes the network portion; the remaining bits are your host space.

Formula: 2^(32 - prefix) = total addresses. AWS reserves 5 per subnet (network address, VPC router, DNS, reserved, broadcast).

CIDR	Total addresses	Usable
/16	65,536	65,531
/20	4,096	4,091
/24	256	251
/28	16	11

To reverse-engineer a prefix from a required host count: round up to the next power of two, subtract the exponent from 32. Need 300 hosts? Next power of two is 512 (2⁹), so prefix = 32 - 9 = /23. Run this before sizing any subnet that will host an autoscaling group or EKS node group.

Start with /16 for the VPC itself. VPC CIDR is difficult to resize after the fact — once you have subnets, peering connections, or Transit Gateway attachments built against it, renumbering becomes a migration project. /16 costs nothing up front and avoids that corner.

Subnet allocation: carving up the VPC

A practical three-AZ production layout from 10.0.0.0/16:

Tier	AZ-a	AZ-b	AZ-c	Size	Typical use
Public	10.0.0.0/24	10.0.1.0/24	10.0.2.0/24	/24	ALB, NAT gateway, bastion
Private/app	10.0.16.0/20	10.0.32.0/20	10.0.48.0/20	/20	EKS nodes, ECS, EC2
Data	10.0.64.0/24	10.0.65.0/24	10.0.66.0/24	/24	RDS, ElastiCache
Reserved	10.0.128.0/17			/17	Future tiers, Transit Gateway, VPN

The jump from /24 in the public tier to /20 in the app tier is intentional. ALBs and NAT gateways consume very few IPs; the app tier is where consumption scales with autoscaling groups, rolling deployments, and pod density.

For EKS specifically: with the VPC CNI, every pod can consume an ENI-backed IP. IP exhaustion is one of the most common EKS production incidents. /20 per AZ for worker subnets is the standard starting point.

The deliberate gaps between tiers (0–2, then 16–48, then 64–66) leave room to insert new tiers later without renumbering anything already deployed.

Route tables: the actual decision maker

A subnet is "public" or "private" because of its route table — not any inherent property of the subnet itself. The table is a list of destination → target rules evaluated by most specific match.

Every route table gets an implicit local route for the full VPC CIDR — this can't be removed, and it's what lets every subnet reach every other subnet inside the VPC by default.

A public subnet route table in Terraform:

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
  tags   = { Name = "rtb-public", Tier = "public" }
}

resource "aws_route" "public_internet" {
  route_table_id         = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.main.id
}

resource "aws_route_table_association" "public_a" {
  subnet_id      = aws_subnet.public_az_a.id
  route_table_id = aws_route_table.public.id
}

Creating the route table does nothing on its own — the association step is what binds it to a subnet and makes routing take effect.

Internet Gateway and the three conditions for inbound access

An IGW is horizontally scaled, redundant, and AZ-agnostic — one per VPC, no capacity to configure. It does two things: 1:1 NAT between public and private IPs (the public IP mapping lives at the IGW, not on the instance — which is why ip addr on an EC2 instance never shows its public IP), and serves as a route table target.

All three of these must be true simultaneously for inbound internet access to work:

The instance has a public or Elastic IP on its ENI.
The subnet's route table has 0.0.0.0/0 → IGW.
Both the security group and the NACL allow the inbound traffic on that port.

Any one missing produces the same symptom: a silent timeout with no obvious pointer to the actual cause. This is where most "why can't I reach my instance" tickets originate.

NAT Gateway: outbound only

A NAT gateway lives in a specific subnet in a specific AZ, performs source NAT for private instances, and has real hourly and per-GB cost. The packet walk for a private instance at 10.0.2.15 requesting a public registry:

Private subnet route table: 0.0.0.0/0 → nat-0abc...
NAT gateway rewrites source to its own Elastic IP + ephemeral port
NAT gateway's public subnet route table: 0.0.0.0/0 → IGW
IGW performs its own separate 1:1 NAT translation

Two distinct NAT translations — easy to collapse into one mental step, but they're separate resources doing separate jobs.

Why it lives in the public subnet: the NAT gateway needs its own route to the IGW, so it must sit in a subnet whose route table already points to the IGW. The private subnet's route table then points 0.0.0.0/0 at the NAT gateway. Two different route tables, two different subnets, one resource bridging them.

HA pattern: one NAT gateway per AZ, each AZ's private subnet routing to the NAT gateway in its own AZ. One NAT gateway for the whole VPC is cheaper but creates a single point of failure — if that AZ has an outage, every private subnet in every other AZ loses outbound internet access.

Cost trap worth auditing: traffic to S3 and DynamoDB from private subnets doesn't need to go through NAT at all if you use VPC Gateway Endpoints. Routing S3 traffic through NAT is billed per GB with no benefit over a free Gateway Endpoint.

Read the Full Article

The summary covers the core mental model. The full article goes deeper on:

The reverse-engineering formula for subnet sizing applied to autoscaling groups and EKS node groups, with the specific IP exhaustion failure mode explained
The full route table example with VPC peering and S3 Gateway Endpoint entries, and why most-specific-match matters for overlapping routes
IGW statelessness and why NACLs require explicit ephemeral port rules (1024–65535) that security groups handle automatically
NAT Gateway connection tracking limits: 55,000 concurrent connections per unique destination, PortAllocationErrors in CloudWatch as the signal, and when to reconsider architecture vs. adding more NAT gateways
The full security group vs. NACL comparison — stateful vs. stateless evaluation, allow-only vs. allow-and-deny, and why the default NACL and default security group behave differently out of the box

👉 AWS VPC Networking Fundamentals — Full Article