DEV Community: Alok Ranjan Daftuar

Queue vs Topic vs Log: Picking the Broker

Alok Ranjan Daftuar — Mon, 02 Nov 2026 10:57:50 +0000

Most broker decisions are made in a ten-minute conversation that ends with "we'll just use Kafka," or from the other side, "SQS is fine, it's cheap." Both sentences treat the broker as a throughput-and-price decision. It isn't. What you are actually choosing is a contract: who owns a message after it is delivered, who tracks how far each consumer has read, and whether yesterday's events still exist tomorrow.

Get the contract wrong and no tuning fixes it. A team that picks a queue and later needs a new service to bootstrap from the last 30 days of order events has nothing to replay. A team that picks Kafka for background jobs ends up owning partition planning, rebalances, and head-of-line blocking to get what a queue provides out of the box.

Three primitives, three contracts

QUEUE (competing consumers)         TOPIC (fan-out)                    LOG (retained stream)

 producer ──▶ [ m3 m2 m1 ] ──┬▶ A    producer ──▶ topic ──┬▶ [sub-A] ▶ A   producer ──▶ [m1 m2 m3 m4 m5 …]
                             └▶ B                         ├▶ [sub-B] ▶ B                  ▲        ▲
   each message goes to ONE            each subscription gets                       consumer X      consumer Y
   consumer, then is deleted           its OWN copy                                 (offset 2)      (offset 5)

	Queue	Topic (pub/sub)	Log
Delivery model	Competing consumers: one handler per message	Fan-out: every subscription gets a copy	Every consumer group reads the whole stream independently
After consumption	Deleted on ack	Deleted per subscription on ack	Retained until retention expires, regardless of reads
Read position owned by	Broker (visibility timeout / lock)	Broker, per subscription	Consumer (offset)
Replay	No	No	Yes, by rewinding offsets
Failure handling	Per-message retry and DLQ	Per-subscription retry and DLQ	None built in: a failing message blocks its partition

The last row is where teams get surprised. In a queue, message 7 failing does not stop message 8. In a log, the consumer's offset means "everything before this is done" — a message that keeps failing stalls its partition unless you build retry topics yourself.

Mapping to AWS and Azure

Primitive	AWS	Azure
Queue	SQS Standard / FIFO	Service Bus queues
Topic	SNS, SNS FIFO	Service Bus topics + subscriptions
Event router	EventBridge	Event Grid
Log	MSK, Kinesis Data Streams	Event Hubs (Kafka endpoint)

Three mapping details decide real designs:

SNS and Service Bus topics are not equivalent. On Azure, a topic subscription is a queue with its own lock, delivery count, and dead-letter sub-queue. On AWS, SNS pushes to a subscriber — the standard answer is SNS → SQS per consumer, which recreates what Service Bus gives natively.

EventBridge and Event Grid are routers, not backbones. Right for cloud/SaaS event reactions. Wrong for a high-volume internal event backbone where you need ordering and replay at a known cost.

Partition count is a design decision, not a tuning knob. Kafka partitions can be increased but not reduced. Event Hubs Basic/Standard partition count is fixed at creation. Size it for how far back a recovering consumer might need to go.

The pattern that covers most systems: topic fan-out into per-consumer queues

When several services need the same event, publish once to a topic and give every consumer its own queue. Each consumer gets isolation, its own DLQ, and independent scaling.

AWS: SNS → SQS with filtering

resource "aws_sqs_queue" "billing" {
  name                       = "billing-order-events"
  visibility_timeout_seconds = 120 # must exceed worst-case handler time
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.billing_dlq.arn
    maxReceiveCount     = 5
  })
}

resource "aws_sns_topic_subscription" "billing" {
  topic_arn            = aws_sns_topic.orders.arn
  protocol             = "sqs"
  endpoint             = aws_sqs_queue.billing.arn
  raw_message_delivery = true # otherwise the body is an SNS JSON envelope
  filter_policy_scope  = "MessageBody"
  filter_policy = jsonencode({
    eventType = ["OrderPlaced", "OrderCancelled"]
  })
}

The non-obvious decisions: raw_message_delivery so consumers read your payload and not an SNS envelope, filter_policy_scope = "MessageBody" so routing works on the payload, and a visibility timeout longer than your slowest handler.

Azure: Service Bus topic with a filtered subscription

resource billing 'Microsoft.ServiceBus/namespaces/topics/subscriptions@2021-11-01' = {
  parent: topic
  name: 'billing'
  properties: {
    lockDuration: 'PT2M'
    maxDeliveryCount: 5
    deadLetteringOnMessageExpiration: true
  }
}

resource billingFilter 'Microsoft.ServiceBus/namespaces/topics/subscriptions/rules@2021-11-01' = {
  parent: billing
  name: 'order-lifecycle'
  properties: {
    filterType: 'SqlFilter'
    sqlFilter: { sqlExpression: 'eventType IN (\'OrderPlaced\', \'OrderCancelled\')' }
  }
}

Service Bus SQL filters evaluate message properties, not the body — the publisher must set eventType as an application property. A subscription without an explicit rule gets a catch-all $Default rule; confirm it's not still attached after deployment or the subscription receives everything.

When you need a log

Reach for a log when retention is the requirement: new consumers must read history, projections must be rebuildable, or stream processing needs ordered per-key state.

kafka-topics.sh --bootstrap-server "$BOOTSTRAP" --create \
  --topic order-events \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000   # 7 days

--partitions 12 is your parallelism ceiling per consumer group: a thirteenth consumer sits idle. replication-factor 3 with min.insync.replicas=2 and acks=all tolerates one broker loss without losing acknowledged writes.

Four questions, in order

#	Question	If yes
1	Must a new or recovering consumer read events from before it existed?	Log
2	Do several independent services need the same event, each at their own pace?	Topic fanning into per-consumer queues
3	Is this work distribution: one handler per message, per-message retry, a DLQ?	Queue
4	Is it a discrete state change from a cloud or SaaS source triggering reactions?	EventBridge / Event Grid

Answers can combine. A common production shape is a log as the system-of-record backbone with queues downstream for consumers that need per-message retry and a DLQ — so one poison message cannot stall a partition.

Read the Full Article

This summary covers the three primitives, the AWS/Azure service mapping, and the decision framework. The full article includes:

Complete Terraform for the SNS → SQS fan-out pattern including the IAM queue policy scoped to the topic ARN
Full Bicep for the Service Bus topic with duplicate detection and filtered subscription
The "exactly-once" section — what SQS FIFO deduplication, Service Bus duplicate detection, and Kafka transactions actually cover (and don't cover)
The archival consumer pattern: how to attach cheap replay insurance from day one when you choose a queue or topic
What the wrong choice looks like in production — queue chosen when a log was needed, log chosen when a queue was needed, topic without a durable buffer

👉 Queue vs Topic vs Log: Picking the Broker — Full Article

Multi-Tenant MCP: Isolation Patterns That Hold Up

Alok Ranjan Daftuar — Fri, 30 Oct 2026 09:26:36 +0000

Every multi-tenant MCP deployment starts with the same instinct: put a tenant_id in the JWT, scope every tool call to it, and call the isolation problem solved. It isn't. The gap between "scoped by claim" and "actually isolated" is exactly where these deployments fail in production — not in testing, where a single code path is exercised, but in production, where a new tool handler forgets the filter, a cache key drops the tenant dimension, or an agent legitimately holds a valid token but aggregates a result that happened to include data from a neighboring tenant.

This article is about what real isolation looks like across three architectural patterns, which layer each one enforces, and how to test the boundary rather than just build it.

Three patterns, not one

Multi-tenant MCP deployments settle into three recognizable shapes. The pattern a platform needs isn't primarily a technical decision — it's downstream of what its most demanding tenant's compliance posture actually requires.

Pattern	Isolation layer	Cost profile	Fits
Shared instance, claim-scoped	Application/data layer	Lowest	Self-serve SaaS, low-sensitivity data
Dedicated compute per tenant	Infrastructure layer (namespace, network, compute)	Highest	SOC 2, HIPAA, FedRAMP tenants
Hybrid — shared compute, per-tenant credential vault	Credential layer + claim-scoping	Moderate	Most production SaaS

Forcing both a self-serve SaaS product and its first enterprise customer onto pattern 1 purely for engineering simplicity is the decision that surfaces as a failed security questionnaire months later, at exactly the point in a sales cycle where it's most expensive to revisit.

Pattern 1: shared instance, claim-scoped

The critical detail the "put tenant_id in the JWT" version of this pattern usually gets wrong is treating claim extraction as authorization. It's only identification — the actual boundary has to be enforced at the data layer, on every path a request can take:

async function handleToolCall(req: McpRequest): Promise<McpResponse> {
  const claims = await verifyToken(req.headers.authorization);
  const tenantId = claims.tenant_id;

  // tenantId passed explicitly — no ambient global a new handler could forget
  const tools = await toolRegistry.listFor(tenantId);
  if (!tools.find(t => t.name === req.params.name)) {
    throw new McpError("tool_not_found", 404); // not 403 — don't confirm existence
  }

  const credentials = await credentialStore.get(tenantId, req.params.name);
  return executeTool(req.params.name, req.params.arguments, credentials, tenantId);
}

Two details matter more than they look. Returning tool_not_found rather than 403 avoids confirming to a caller that a tool exists at all — the difference between an isolation boundary and one that leaks information about what's on the other side. And passing tenantId explicitly through every function call, rather than reading it from a request-scoped global, is deliberate: an ambient "current tenant" is exactly the kind of implicit state that a new tool handler, added six months later by someone who didn't write this code, can forget to consult.

This pattern also depends on the MCP spec's OAuth 2.1 + PKCE baseline and Resource Indicators (RFC 8707) — which closes the specific failure mode where a token issued for one MCP server gets replayed against a different one. None of that is optional hardening; it's the baseline the pattern depends on to be a security boundary at all.

Pattern 2: dedicated compute per tenant

For tenants under SOC 2, HIPAA, or FedRAMP obligations, claim-scoping inside a shared process is rarely sufficient — the compliance requirement is usually for a demonstrable infrastructure-level boundary. This is where namespace-and-labeling conventions stop being hygiene and start being the actual isolation mechanism:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-cross-tenant
  namespace: mcp-tenant-acme
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              tenant: acme
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              tenant: acme
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns

The NetworkPolicy is the actual isolation enforcement: pods in mcp-tenant-acme can only reach other pods labeled with the same tenant, plus an explicit DNS exception. A ResourceQuota alongside it addresses the noisy-neighbor problem — without it, one tenant's traffic spike can starve every other tenant even inside dedicated namespaces. The pattern only holds if quota, network policy, and RBAC are applied consistently across every tenant namespace — which is exactly the kind of thing worth generating from a template rather than hand-writing per tenant as the customer count grows.

Sandboxing the execution layer

Namespace-level isolation stops one tenant's pod from reaching another over the network. It does nothing about a single pod's own execution boundary. Standard containers share the host kernel — a RuntimeClass backed by gVisor interposes a user-space kernel so agent-executed code never touches host syscalls directly:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

Worth being specific about what gVisor buys versus what it doesn't. A runsc-backed pod still shares the Kubernetes control plane and — absent the NetworkPolicy above — the node's network stack. The syscall interception layer closes one specific class of escape (a kernel exploit reached through a container's syscall surface); it says nothing about a request that legitimately reaches the pod over the network and abuses an application-level bug once inside. Pattern 2's NetworkPolicy and this section's RuntimeClass are answering two different questions — who can reach this pod, and what can code running inside this pod reach past the kernel — and a deployment that only implements one of them has a boundary with a specific, known hole in it.

The hybrid most production platforms actually run

In practice, the pattern that shows up most in production is shared compute for the MCP server fleet combined with a per-tenant credential vault — so a compromise of the shared fleet doesn't automatically compromise every tenant's downstream credentials at once:

async function getCredential(tenantId: string, toolName: string): Promise<Credential> {
  const vaultPath = `tenants/${tenantId}/tools/${toolName}`;
  return vaultClient.read(vaultPath); // short-lived, tenant-scoped token
}

The credential vault (HashiCorp Vault, AWS Secrets Manager with per-tenant IAM policies, or Azure Key Vault with per-tenant access policies) becomes the actual blast-radius boundary. A bug in the shared server's tenant-scoping logic is a serious incident, but it isn't automatically a credential-exfiltration incident for every tenant simultaneously — because the credentials were never loaded into a shared, unscoped in-memory store to begin with. This is the pattern worth defaulting to unless a specific tenant's compliance requirement forces pattern 2's dedicated compute.

Testing the boundary, not just building it

An isolation pattern that hasn't been tested for cross-tenant leakage is a claim, not a guarantee:

TOKEN_A=$(get_token_for_tenant acme)
TOKEN_B=$(get_token_for_tenant globex)

TOOL_ID=$(curl -s -X POST https://mcp.example.com/mcp \
  -H "Authorization: Bearer $TOKEN_A" \
  -H "Mcp-Method: tools/call" -H "Mcp-Name: create-record" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"create-record"}}' \
  | jq -r '.result.id')

# Tenant B attempting to read Tenant A's record must fail
curl -s -X POST https://mcp.example.com/mcp \
  -H "Authorization: Bearer $TOKEN_B" \
  -H "Mcp-Method: tools/call" -H "Mcp-Name: get-record" \
  -d "{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"tools/call\",\"params\":{\"name\":\"get-record\",\"arguments\":{\"id\":\"$TOOL_ID\"}}}" \
  | jq -e '.error.code == "tool_not_found" or .error.code == "not_found"'

Wiring this into CI means an accidental regression in tenant-scoping logic — a new tool handler that forgets the filter, a cache key that drops the tenant dimension — fails a build instead of surfacing as a support ticket from a customer who noticed data that wasn't theirs.

Read the Full Article

This summary covers the three patterns and their core mechanics. The full article includes:

Why tool_not_found vs 403 matters for information leakage at the isolation boundary
The full ResourceQuota + NetworkPolicy + RBAC combination for dedicated-compute tenants
When to use gVisor vs Kata Containers vs Firecracker, and how to match sandboxing depth to actual tool risk
The OAuth 2.1 + Resource Indicators baseline that pattern 1 depends on to be a real security boundary
Why the hybrid pattern captures most of pattern 2's blast-radius protection at close to pattern 1's cost

👉 Multi-Tenant MCP: Isolation Patterns That Hold Up — Full Article

DNS in AWS: Route 53 Resolver, Private Hosted Zones, and Hybrid Resolution

Alok Ranjan Daftuar — Tue, 27 Oct 2026 07:44:09 +0000

Three earlier articles in the AWS Network Architecture series each hit the same wall and moved past it: VPC Endpoints needed private_dns_enabled to work transparently. VPC Peering noted that DNS resolution across a peering connection needs its own explicit setting. Multi-Account Networking flagged that Transit Gateway spoke accounts don't automatically share DNS resolution scope. This article is the piece all three were pointing toward.

Connectivity at the routing layer and name resolution are two separate problems. It's entirely possible — common, even — to have flawless routing and completely broken DNS, or vice versa. Both layers have to be built correctly, and this article covers the DNS side end to end.

The Default: What a VPC Resolves Out of the Box

Every VPC gets a built-in DNS resolver at its base CIDR plus two — for 10.0.0.0/16, that's 10.0.0.2. It's provided automatically, no provisioning required.

# From inside an EC2 instance
cat /etc/resolv.conf
# nameserver 10.0.0.2

nslookup s3.ap-south-1.amazonaws.com 10.0.0.2

This default resolver handles three things without any additional configuration: public DNS, AWS service endpoints, and any private hosted zone associated with that specific VPC. That third one is where most of this article lives — it works cleanly for a single VPC, and requires deliberate extra infrastructure the moment more than one VPC needs to share the same private namespace.

Private Hosted Zones: DNS Scoped to a VPC

A Route 53 private hosted zone only resolves for VPCs it's explicitly associated with — it never appears in public DNS. This is the mechanism behind private_dns_enabled on Interface endpoints: AWS creates a private hosted zone, associates it with the endpoint's VPC, and that association makes the service's standard public name resolve to the endpoint's private IP.

resource "aws_route53_zone" "internal" {
  name = "internal.example.com"

  vpc {
    vpc_id = aws_vpc.main.id
  }
}

resource "aws_route53_record" "app_db" {
  zone_id = aws_route53_zone.internal.zone_id
  name    = "db.internal.example.com"
  type    = "A"
  ttl     = 300
  records = [aws_db_instance.main.address]
}

An instance outside the associated VPC gets NXDOMAIN — the zone simply doesn't exist from that vantage point, by design.

Cross-Account Zone Association

A private hosted zone can be associated with VPCs in other AWS accounts. Cross-account association requires an explicit authorization step from the zone-owning account first — a deliberate security gate so no account can silently attach to another account's internal DNS namespace:

# Zone-owning account: authorize the spoke VPC
aws route53 create-vpc-association-authorization \
  --hosted-zone-id Z0ABC123DEF \
  --vpc VPCRegion=ap-south-1,VPCId=vpc-0spokeA1234

# Spoke account: complete the association
aws route53 associate-vpc-with-hosted-zone \
  --hosted-zone-id Z0ABC123DEF \
  --vpc VPCRegion=ap-south-1,VPCId=vpc-0spokeA1234

This two-step authorize-then-associate pattern fills the DNS gap that Transit Gateway routing alone doesn't provide in a centralized multi-account model.

Route 53 Resolver: For Anything Crossing Outside AWS

Zone association solves internal-name resolution within AWS. It doesn't solve resolving AWS-side private DNS names from outside AWS, or resolving on-prem/Azure DNS names from inside a VPC. Both directions need Route 53 Resolver endpoints.

An inbound endpoint gives external resolvers a private IP inside the VPC to send queries to. An outbound endpoint with forwarding rules lets VPC-side resolvers forward specific domain queries out to an external DNS server:

resource "aws_route53_resolver_endpoint" "outbound" {
  name      = "resolver-outbound-to-azure"
  direction = "OUTBOUND"

  security_group_ids = [aws_security_group.resolver.id]

  ip_address { subnet_id = aws_subnet.shared_a.id }
  ip_address { subnet_id = aws_subnet.shared_b.id }
}

resource "aws_route53_resolver_rule" "to_azure" {
  domain_name          = "corp.internal.azure.example.com"
  rule_type            = "FORWARD"
  resolver_endpoint_id = aws_route53_resolver_endpoint.outbound.id

  target_ip {
    ip = "10.20.0.10"  # Azure private DNS resolver IP
  }
}

resource "aws_route53_resolver_rule_association" "spoke_a" {
  resolver_rule_id = aws_route53_resolver_rule.to_azure.id
  vpc_id           = "vpc-0spokeA1234"
}

The rule association step matters as much as the rule itself — a Resolver rule that exists but isn't associated with a given VPC has no effect there.

Sharing Resolver Rules Across Accounts via RAM

Resolver rules follow the same centralized-ownership pattern as Transit Gateway sharing: a network account creates and owns the outbound endpoint and forwarding rules, then shares them to spoke accounts via AWS Resource Access Manager:

resource "aws_ram_resource_share" "resolver_rules" {
  name                      = "ram-resolver-rules-to-azure"
  allow_external_principals = false
}

resource "aws_ram_principal_association" "org_share" {
  principal          = data.aws_organizations_organization.this.arn
  resource_share_arn = aws_ram_resource_share.resolver_rules.arn
}

Sharing to the whole Organization ARN means a newly created spoke account can associate with the existing forwarding rule immediately, without a manual RAM step per new account.

The Failure Checklist

DNS-across-boundaries failures share a common symptom — a name that resolves fine from one vantage point and fails from another. Check in this order:

Is the private hosted zone actually associated with the VPC making the query?
For cross-account association, was the authorization step completed on the zone-owning side?
For Resolver rules, is the rule associated with the specific VPC — not just created?
Is a custom DHCP option set overriding the VPC's default resolver?
For hybrid resolution, is the underlying VPN/network path actually up?

Read the Full Article

The full post covers additional depth on:

The complete side-by-side mechanism summary table (default resolver vs. zone association vs. Resolver endpoints)
Resolver query logging with Terraform — turning DNS debugging from guesswork into a direct log lookup
Why DNSSEC is orthogonal to everything in this article and what it's actually for
How this connects to the hybrid Azure/AWS migration scenario from the Migration Playbook series — both inbound and outbound resolution directions needed simultaneously during a phased migration

👉 DNS in AWS: Route 53 Resolver, Private Hosted Zones, and Hybrid Resolution — Full Article

CIDR Planning for Azure-to-AWS Migrations: Reconciling Two Address Spaces That Were Never Designed Together

Alok Ranjan Daftuar — Fri, 23 Oct 2026 05:55:50 +0000

CIDR planning for a migration is not the same exercise as CIDR planning for a greenfield VPC. In a greenfield environment you pick a clean /16, carve it into tiers, and you're done. In a migration, the target VPC has to coexist with — and eventually replace — a source Azure VNet that was sized and addressed by someone solving a completely different problem, possibly years earlier. The Azure side's CIDR is a fixed constraint, not a design choice. Everything downstream depends on knowing it accurately before a single AWS resource is provisioned.

This post is part of the AWS Migration Playbook series. It picks up a thread left open in the MGN architecture article — the staging subnet's positioning relative to the rest of the target VPC — and connects it to the broader CIDR reconciliation problem that trips up nearly every real migration at some point.

Audit the Azure Side First

The most common cause of migration CIDR problems is starting the AWS-side design before fully knowing the Azure-side address space. Run this across every VNet in scope — not just the one hosting the current wave's servers:

az network vnet list --query '[].{Name:name, CIDR:addressSpace.addressPrefixes}' -o table
az network vnet subnet list --vnet-name <vnet-name> --resource-group <rg-name> \
  --query '[].{Name:name, CIDR:addressPrefix}' -o table

If the organization has grown through acquisition or multiple teams provisioning Azure independently, don't assume VNet CIDRs were coordinated centrally. A VNet audited late — after the target VPC's early tiers are already locked in — might turn out to need a range that only fits in whatever's left over.

Why Overlap Matters Even Without Hybrid Connectivity

The "we're leaving Azure behind anyway" assumption holds right up until one of these scenarios appears — and they appear in nearly every real migration:

Rollback plans that require the Azure source and AWS target to be reachable from each other during a validation window, via VPN or equivalent — unusable if CIDRs overlap, full stop.
Phased migrations where some services remain on Azure while others move to AWS, and the two environments need to communicate during the transition.
Future merges with another AWS Organization or cloud footprint that happens to use the same range — unlikely to be on anyone's mind during the current migration, but a recurring cause of expensive re-IP projects years later.

Treat overlap avoidance as a hard requirement for every migration, not just ones with a known hybrid connectivity need at planning time.

Building the Reconciliation Table

Once every Azure VNet's CIDR is documented, build an explicit table of every range in use across both clouds:

Environment	CIDR	Status
Azure VNet: prod-eastus	10.20.0.0/16	Source, in use
Azure VNet: dev-eastus	10.21.0.0/16	Source, in use
Azure VNet: shared-services	10.30.0.0/16	Source, staying on Azure
AWS VPC: migration-target (proposed)	10.0.0.0/16	Proposed — needs validation

The proposed AWS range only gets marked valid once it's confirmed non-overlapping against every row — including VNets that aren't being migrated but will remain on Azure indefinitely. A range colliding with a permanently-staying VNet is just as blocking as one colliding with a VNet you're actively migrating away from.

The MGN Staging Subnet Deserves Its Own Allocation

The staging subnet MGN uses to receive replicated data should be a deliberate, separate row in the tier table — not squeezed into whatever subnet has spare capacity:

10.0.0.0/16   vpc-migration-target
├── 10.0.0.0/24      public
├── 10.0.16.0/20     private/app
├── 10.0.64.0/24     data
├── 10.0.100.0/24    mgn-staging          ← new, migration-specific

Two reasons this matters: staging infrastructure is temporary — present only for active migration waves and safe to fully decommission after cutover. Mixing it into a permanent tier makes cleanup harder to reason about. And a dedicated subnet makes it trivial to apply a tightly scoped security group and NACL specific to replication traffic, rather than inheriting the public tier's broader ruleset.

A /24 (251 usable addresses) comfortably supports a wave of dozens of concurrent source servers. For unusually large waves, size explicitly against planned concurrent server count rather than assuming /24 is always sufficient.

Reserve Headroom for Later Waves

For multi-wave migrations, the target VPC needs headroom for VNets that haven't been audited yet at the time the first wave's CIDR plan is locked in:

10.0.0.0/16   vpc-migration-target
├── 10.0.0.0/24      public                    (wave 1)
├── 10.0.16.0/20     private/app — wave 1
├── 10.0.64.0/24     data — wave 1
├── 10.0.100.0/24    mgn-staging
├── 10.0.128.0/18    RESERVED — wave 2 (finance VNet, not yet audited)
├── 10.0.192.0/18    RESERVED — future waves

Reserve entire unused CIDR ranges, not just gaps between adjacent subnets. "Whatever's left over" is a much worse position to negotiate from than reserving room deliberately from the start.

DNS Coexistence During the Transition Window

CIDR reconciliation solves whether IP ranges can coexist — it doesn't solve whether names resolve correctly across both environments. For phased migrations where wave-one AWS services need to be reachable by private DNS name from wave-two Azure services, you need deliberate DNS forwarding:

Azure Private DNS zones need conditional forwarding rules pointing at a resolver reachable in the AWS VPC (Route 53 Resolver inbound endpoint or the VPC's .2 resolver address)
The reverse direction needs equivalent configuration on the AWS side

Skipping this doesn't fail loudly — it fails as intermittent-looking connectivity issues that look like application bugs rather than a missing DNS forwarding rule.

Validate Before the First Server Replicates

Before starting MGN replication for a wave's first server, validate empirically:

# From an Azure VM — confirm no undocumented overlap or peering
ip route show | grep -E "10\.(0|20|21|30)\."

# From AWS — confirm staging subnet CIDR matches the plan
aws ec2 describe-subnets --filters "Name=tag:Name,Values=mgn-staging" \
  --query 'Subnets[*].{Subnet:SubnetId, CIDR:CidrBlock}' --output table

This catches two recurring classes of mistake cheaply: a route on the Azure side hinting at a connection the audit missed, and a staging subnet provisioned with a different CIDR than the plan called for — easy to happen when a subnet is created manually or from a stale template.

Read the Full Article

The full post covers additional depth on:

The complete pre-wave CIDR checklist — five explicit gates to confirm before locking in a target VPC CIDR for any new wave
Tier-to-tier mapping between Azure VNet subnets and AWS VPC tiers, including the AzureBastionSubnet decision
How this connects to the kernel fix and serial console recovery incidents from earlier in the series — the same "should have caught this before the maintenance window" cost pattern
Why the overlap problem compounds specifically in phased migrations vs. lift-and-shift

👉 CIDR Planning for Azure-to-AWS Migrations — Full Article

Drift, Self-Heal, and Failure Modes: When ArgoCD's Reconciliation Loop Fights Something Else

Alok Ranjan Daftuar — Wed, 21 Oct 2026 07:02:02 +0000

Auto-sync and self-heal are usually presented as unambiguous wins. They are — right up until something else in the cluster has a legitimate reason to change the same resource ArgoCD thinks it owns exclusively. When that happens, the result isn't a clear error. It's a quiet oscillation that looks like ArgoCD misbehaving, when it's actually doing exactly what it was told.

This article maps the three most common shapes of that collision, why each one happens, and the precise fix for each.

What Self-Heal Actually Does

ArgoCD's OutOfSync status means live cluster state doesn't match what's declared in Git. With selfHeal: true, ArgoCD doesn't just refuse to accept the drift — it actively overwrites it back to the Git-declared state on every reconciliation pass.

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

That's the right default for almost everything a Deployment touches directly. The failure modes below all share one shape: a resource ArgoCD considers itself the sole owner of is also, legitimately, being modified by something else that has no idea ArgoCD exists.

HPA vs. Self-Heal: The Replica Count Fight

This is the most common collision. A HorizontalPodAutoscaler manages spec.replicas dynamically. Git, in a naive GitOps setup, also declares spec.replicas. The result:

Load spikes ──► HPA scales Deployment to 8 replicas
                        │
                        ▼
        ArgoCD reconciliation runs
                        │
                        ▼
        Live replicas (8) != Git-declared replicas (3)
                        │
                        ▼
        selfHeal reverts Deployment back to 3 replicas
                        │
                        ▼
        HPA scales back to 8
                        │
                        └──── repeats every reconciliation interval

At a 3-minute reconciliation interval during a traffic spike, this produces roughly ten full oscillation cycles over 30 minutes — the service spends a meaningful fraction of the spike window under-provisioned, and the sawtoothing replica count reads as ArgoCD being broken rather than working exactly as configured.

The fix is ignoreDifferences scoped to that field:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-service
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

ArgoCD stops treating spec.replicas as part of what it reconciles for this resource. The HPA becomes the sole owner of that field. Worth auditing across every HPA-managed Deployment in a fleet before selfHeal: true is enabled fleet-wide — not reactively after the first incident.

Sync Waves and Partial Failure

Sync-wave ordering is a sequencing guarantee, not a transactional one. A wave-0 database migration Job that fails partway through doesn't roll back what wave-1 already applied — the Deployment's Pods may already be running against a schema mid-migration before ArgoCD retries wave 0.

For anything where partial application genuinely can't be tolerated, a PreSync hook with hook-failure-policy: Abort is the safer pattern:

metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-failure-policy: Abort

Abort stops the entire sync — including later waves — the moment the hook fails, rather than leaving later waves to apply against a database that's still mid-migration. Self-heal's retry behavior on a failed wave is useful for transient failures; it's the wrong tool when partial application has real consequences.

Operators Mutating ArgoCD-Owned Resources

The third shape is structurally identical to the HPA case: a Kubernetes operator — a cert-manager controller, a service mesh sidecar injector, a mutating admission webhook — modifying a field on a resource ArgoCD also tracks. ArgoCD sees a live-vs-Git diff, self-heal reverts it, the operator re-mutates it, and the oscillation repeats.

The fix is identical — ignoreDifferences scoped to the contested field:

spec:
  ignoreDifferences:
    - group: ""
      kind: Pod
      jsonPointers:
        - /spec/containers/0/resources

The harder part isn't the fix — it's noticing the collision exists. Unlike the HPA case, which announces itself through a visibly oscillating replica count, a webhook fight over a less visible field presents as intermittent OutOfSync with no obvious pattern. When an Application shows unexplained intermittent drift, check kubectl get events on the affected resource and the list of registered mutating webhooks before assuming the drift is a Git or ArgoCD problem.

Decision Framework

Symptom	Likely cause	Fix
Replica count oscillates on a fixed interval	HPA fighting a Git-declared `spec.replicas`	`ignoreDifferences` on `/spec/replicas`
Downstream Application fails with no clear cause after a transient upstream wave failure	Sync-wave retry without transactional guarantees	`PreSync` hook with `hook-failure-policy: Abort`
`OutOfSync` appears and clears with no clear pattern on a specific field	Mutating webhook or operator modifying a field ArgoCD tracks	`ignoreDifferences` scoped to the contested field
Unclear which field is contested	Diff noise	`argocd app diff <app-name>` against live state to identify the exact field before scoping `ignoreDifferences`

Read the Full Article

The full article covers:

The precise mechanism of selfHeal: true and why it's correct by default
The HPA oscillation failure at real numbers — 10 oscillation cycles over a 30-minute traffic spike, what it looks like on a dashboard, and why it reads as ArgoCD being broken
The sync-wave partial failure scenario in detail — a backfill Job, a lock timeout, and application code reading nulls it wasn't written to handle
The service mesh sidecar injector collision — why it presents as intermittent OutOfSync with no obvious trigger, and how to trace it to the actual controller
Series takeaways across all six articles — the thread connecting hub-and-spoke architecture, ApplicationSets, CI handoff, secrets management, progressive delivery, and failure modes into one coherent GitOps discipline

👉 Drift, Self-Heal, and Failure Modes — Full Article

Deploying MCP Servers on Kubernetes

Alok Ranjan Daftuar — Fri, 16 Oct 2026 06:30:46 +0000

Most MCP servers start life as a local, stdio-based process — the client spawns the server as a child process, messages flow over stdin and stdout. The instinct once that server needs to serve more than one client is to containerize it and expose a port. That instinct produces a container that starts, passes a naive health check, and then does nothing useful.

A process reading from stdin and writing to stdout inside a container has no listener, no port, nothing for a Kubernetes Service to route traffic toward. The fix isn't a networking change — it's a transport change in the server itself, from stdio to Streamable HTTP, before Kubernetes enters the picture at all.

The image

The Dockerfile is unremarkable by design — the interesting decisions happen in the manifests, not the image. What matters is that the process binds to HTTP transport, not stdio, as its entrypoint:

FROM node:22-slim AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
RUN npm run build

FROM node:22-slim
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
ENV MCP_TRANSPORT=streamable-http
ENV MCP_HTTP_ADDR=0.0.0.0:8080
ENV MCP_HTTP_ENDPOINT=/mcp
EXPOSE 8080
USER node
ENTRYPOINT ["node", "dist/server.js"]

The three MCP_* variables mirror the flags production-grade MCP server implementations expose for choosing transport mode at startup. Transport, bind address, and endpoint path all need to be explicit and set at deploy time, not hardcoded into the binary.

The Deployment — probes that understand the workload

An MCP server mid-tool-call is not the same thing as a hung process. The probes need to say so explicitly:

livenessProbe:
  httpGet:
    path: /health          # answers: is the process itself alive?
    port: 8080
  periodSeconds: 15
  timeoutSeconds: 5
  failureThreshold: 3      # ~45s of true unresponsiveness before restart

readinessProbe:
  httpGet:
    path: /ready            # answers: can this pod accept new work right now?
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

/health should do nothing more than confirm the process's event loop is running — never block on the status of an in-progress tool call. /ready reflects capacity: a pod mid-reasoning can report itself not-ready for new work without being treated as dead.

Two other details in the Deployment matter: terminationGracePeriodSeconds: 45 (the default 30s is a poor fit for a process that might be mid-tool-call during a rolling update), and resource requests set deliberately modest relative to limits — MCP servers spend most of their wall-clock time waiting on outbound I/O, not computing locally.

The Service — the payoff of stateless transport

apiVersion: v1
kind: Service
metadata:
  name: mcp-server
spec:
  selector:
    app: mcp-server
  ports:
    - port: 443
      targetPort: 8080
      name: https

No session affinity. No cookie-based stickiness annotation. Nothing beyond ordinary load balancing. This is the direct payoff of the July 2026 MCP spec removing the protocol-level session — any request can land on any pod, so the Service stays genuinely simple.

EKS: IRSA for short-lived credentials

An MCP server calling AWS APIs needs credentials. The deployment-time decision that matters most is avoiding a long-lived static credential baked into the image or mounted as a Secret. IAM Roles for Service Accounts (IRSA) binds a Kubernetes ServiceAccount to an IAM role via OIDC federation:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mcp-server
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/mcp-server-role

The IAM role should be scoped narrowly to whatever the server's tools actually need — an MCP server's tool surface is exactly the kind of thing that grows without anyone revisiting the IAM policy that backs it.

AKS: Workload Identity

Workload Identity is AKS's equivalent — it federates a Kubernetes ServiceAccount token with an Entra ID application:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mcp-server
  annotations:
    azure.workload.identity/client-id: "11111111-2222-3333-4444-555555555555"
  labels:
    azure.workload.identity/use: "true"

The azure.workload.identity/use: "true" label must be present on both the ServiceAccount and the pod template — a pod missing that label silently falls back to no identity at all rather than failing loudly, which makes it a frustrating thing to debug after the fact.

Autoscaling — a safe floor, not the ideal signal

CPU utilization doesn't track I/O-bound tool-call concurrency well, but a deployment needs something running from day one:

spec:
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300

The asymmetric stabilization windows are deliberate: scale up quickly (30s) to protect against bursts of concurrent tool-calling tasks, scale down slowly (300s) to avoid flapping. This will under-react to the specific load pattern MCP servers produce, but it's a safe default while a better signal is put in place.

Rolling updates without dropping in-flight work

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

maxUnavailable: 0 guarantees the rollout never reduces capacity below the current replica count — a new pod must pass its readiness probe before the corresponding old pod receives SIGTERM and begins the 45-second grace period. The difference between a rollout that's invisible to callers and one that produces a visible error-rate blip every time code ships.

Verifying the deployment

A request that exercises the actual transport confirms the deployment is correctly speaking the current specification:

curl -s -X POST https://mcp.example.com/mcp \
  -H "Content-Type: application/json" \
  -H "Mcp-Method: tools/list" \
  -H "Mcp-Name: mcp-server" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'

A correctly deployed 2026-07-28-compliant server returns the tool list with no Mcp-Session-Id in the response headers and no initialize call required beforehand. If a session header shows up, the server (or a proxy in front of it) is still running the older stateful transport path. Worth catching in verification rather than discovering under load — and worth running as a smoke test on every deploy, not just at initial deployment.

Read the Full Article

The summary covers the core patterns. The full article goes deeper on:

The complete Deployment YAML with all fields explained — resource requests, grace period, and why each value is what it is
Why credential federation (IRSA and Workload Identity) matters more than it looks like it should — and the measured ecosystem data on static credential exposure in open-source MCP servers
Namespace and labeling conventions worth setting at initial deployment, before isolation requirements force a full manifest rewrite
The rolling update strategy in full context, and why maxUnavailable: 0 is the right default for this workload class

👉 Deploying MCP Servers on Kubernetes — Full Article

Multi-Account AWS Networking: Shared VPC, RAM, and the Hub-and-Spoke Model

Alok Ranjan Daftuar — Thu, 15 Oct 2026 06:25:09 +0000

Once an AWS Organization crosses a handful of accounts, the question stops being how to connect VPCs and becomes who owns the network at all. Two genuinely different models answer that question differently — Shared VPC and centralized Transit Gateway — and picking the wrong one early is expensive to unwind later.

This post maps both models: what each centralizes, what each delegates, where each breaks down, and how to choose between them based on what your Organization actually needs.

Model 1: Shared VPC

Shared VPC uses AWS Resource Access Manager to let one account (typically a central network account) own a VPC and share specific subnets with other accounts. Those accounts launch resources directly into the shared subnets — EC2, RDS, Lambda in VPC mode — without creating a VPC of their own.

# Network account: share a subnet with a spoke account
resource "aws_ram_resource_share" "vpc_share" {
  name                      = "ram-shared-vpc-subnets"
  allow_external_principals = false
}

resource "aws_ram_resource_association" "app_subnet" {
  resource_arn       = aws_subnet.shared_app_a.arn
  resource_share_arn = aws_ram_resource_share.vpc_share.arn
}

resource "aws_ram_principal_association" "app_account" {
  principal          = "222233334444"
  resource_share_arn = aws_ram_resource_share.vpc_share.arn
}

The network account retains ownership of route tables, NACLs, Internet Gateway, and NAT gateways. Spoke accounts cannot modify any of these — they can only launch resources into the subnets they've been given. Security groups are the one exception: each account creates and manages its own security groups for the resources it launches, even in a shared subnet.

This split — centralized routing and NACLs, decentralized security groups — defines how Shared VPC feels operationally: the network team owns "can traffic reach this subnet at all," and application teams own "can this specific instance be reached."

Where it earns its complexity: accounts that genuinely need to behave as if they're in the same network — shared load balancers, a common data tier, low-latency service-to-service calls. No Transit Gateway hop, no per-attachment cost.

Where it breaks down: everything sharing the VPC shares the same blast radius. A misconfigured NACL in the network account affects every spoke simultaneously. Two spoke accounts can reach each other's resources over the VPC's local route by default — isolation requires deliberate security group discipline that Transit Gateway's route table segmentation gives you structurally.

Model 2: Centralized Transit Gateway (Hub-and-Spoke)

Each account owns its own full VPC. A central network account owns a Transit Gateway shared via RAM — every spoke account attaches its VPC to the shared TGW.

# Network account: share TGW with the entire Organization
resource "aws_ec2_transit_gateway" "central" {
  description                     = "tgw-org-central"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"
}

resource "aws_ram_principal_association" "org_share" {
  principal          = data.aws_organizations_organization.this.arn
  resource_share_arn = aws_ram_resource_share.tgw_share.arn
}

Sharing to the Organization ARN rather than individual account IDs means any new account added to the Organization can attach to the TGW without a manual RAM association step — important for Organizations still adding accounts regularly.

Each spoke account attaches independently:

resource "aws_ec2_transit_gateway_vpc_attachment" "spoke" {
  subnet_ids         = [aws_subnet.app_private_a.id, aws_subnet.app_private_b.id]
  transit_gateway_id = "tgw-0abc123central"
  vpc_id             = aws_vpc.spoke.id
}

The network team's authority is limited to the TGW's own route tables. Everything inside each spoke VPC — CIDRs, route tables, security groups, NACLs — is fully owned by that account's team.

The actual selling point: TGW route table associations give you structural segmentation. A production route table, a non-production route table, and a shared-services route table each control a completely different set of reachable attachments. A spoke account has no ability to affect that segmentation from its side. This is the property Shared VPC can't cleanly replicate.

The cost: every spoke VPC is a full VPC — its own CIDR allocation, its own NAT gateway, its own VPC endpoints. For an Organization with many small accounts, this multiplies infrastructure cost significantly compared to a single shared VPC.

The Comparison

	Shared VPC	Centralized Transit Gateway
VPC ownership	One VPC, centrally owned	Each account owns its own VPC
Segmentation between accounts	Requires security group discipline	Native, via TGW route table association
CIDR planning	One VPC's CIDR to manage	Every spoke needs non-overlapping CIDR
Cost	Lower — one NAT/endpoint fleet	Higher — NAT/endpoints multiplied per spoke
Best fit	Many small teams needing shared infrastructure access	Organizations needing strong trust-boundary segmentation

A Hybrid Pattern Worth Knowing

Keep each application account on its own full VPC connected via Transit Gateway, but put genuinely shared infrastructure — a central logging aggregator, an internal artifact registry, shared Active Directory — in its own dedicated VPC also attached to the same TGW. Every spoke gets low-friction access through normal TGW routing, gated by the same route table segmentation used for everything else. No ownership ambiguity about which application account "hosts" the shared infrastructure.

DNS Across Account Boundaries

Routing connectivity doesn't automatically mean private DNS resolution works across accounts.

For Shared VPC this is largely a non-issue — spoke accounts are in the same VPC, so standard VPC DNS resolution covers everything uniformly.

For centralized TGW, each spoke has its own DNS scope. Cross-account private hosted zone resolution needs to be built deliberately — either through Route 53 Private Hosted Zone association across accounts (requires an authorization step from the zone-owning account) or through a centralized Route 53 Resolver with inbound/outbound endpoints in the shared services VPC. This is one of the more fiddly parts of the hub-and-spoke model and worth treating as its own planning task during initial Organization network design.

Multi-Account-Specific Failure Modes

Beyond the standard route table/security group/NACL checklist:

RAM share exists but principal association is missing or not accepted — the resource simply doesn't appear in the spoke account with no error, just an absence
SCPs blocking cross-account resource use independent of RAM being correctly configured — produces an IAM access denied error, easy to spend time debugging the network layer when the block is a policy layer above it
CIDR overlap discovered only at TGW attachment time — two spoke accounts provisioned independently can end up with overlapping ranges that only become visible when both try to attach and their routes conflict

Read the Full Article

This summary covers both models, the comparison table, the hybrid pattern, DNS considerations, and the key failure modes. The full article includes:

Complete Terraform for Shared VPC — VPC, subnets, RAM share, principal association
Complete Terraform for centralized TGW — TGW with disabled default route tables, Organization-wide RAM share, spoke attachment
Why disabling default TGW route table association matters for segmentation
The migration-in-progress angle — how the account structure decided before the first MGN wave determines which model is even on the table later
Full troubleshooting section for multi-account-specific failure modes

👉 Multi-Account AWS Networking: Shared VPC, RAM, and the Hub-and-Spoke Model — Full Article

Recovering an Azure VM Boot Loop with Serial Console, Mid-Migration

Alok Ranjan Daftuar — Mon, 12 Oct 2026 08:10:26 +0000

Installing a new kernel, running update-grub, and purging the old kernel in the same maintenance window — before rebooting to confirm the new one actually boots — is a sequence that feels safe right up until the VM never comes back up. With no fallback GRUB entry left and no SSH, no RDP, nothing reachable, the only path back is something that doesn't depend on the OS network stack at all. This is what that recovery looks like in practice, and what the root cause turned out to be.

What Azure Serial Console actually is

Azure Serial Console gives direct access to the VM's serial port (COM1) through the Azure portal, completely independent of the guest OS's network stack. This is the critical property: it works when SSH, RDP, and every other normal access path are unreachable — including when the VM is stuck in a boot loop and never finishes booting at all.

Two prerequisites worth confirming before doing any kernel-level work, not during an incident:

Boot diagnostics must be enabled on the VM — verify this explicitly on any VM you're about to do kernel work on, not assumed
The portal account needs Virtual Machine Contributor or equivalent — read-only VM visibility is not enough for serial console access

How the boot loop happened

The kernel swap sequence was: install linux-generic, run update-grub, purge linux-azure — all in the same session, without an intervening reboot. The assumption was that a successful package install and a clean update-grub run meant the new kernel would boot. That assumption was wrong: the linux-generic kernel had a dependency on an initramfs configuration that hadn't regenerated correctly for the Azure Hyper-V environment, and the VM never came back after the reboot.

With linux-azure already purged, there was no fallback GRUB entry. The VM sat in a continuous restart loop — visible in the Azure portal, but with no SSH, no RDP, nothing reachable.

The recovery sequence

Connecting via serial console and pressing Esc during boot interrupts GRUB's countdown and lands on the menu:

GNU GRUB  version 2.06

*Ubuntu
 Advanced options for Ubuntu

With a fallback kernel still present, selecting it directly is the fastest path back. In this case, linux-azure had already been purged, so only linux-generic entries remained — including a recovery mode option that also failed, since the problem was in the initramfs itself.

The next step: drop to the GRUB command line and confirm what's actually on disk:

grub> ls
(hd0) (hd0,gpt2) (hd0,gpt1) (hd0,gpt15)

grub> ls (hd0,gpt2)/boot
grub.cfg  vmlinuz-6.8.0-1015-generic  initrd.img-6.8.0-1015-generic  ...

The kernel and initramfs files were both present — ruling out a missing-file problem and pointing toward a boot configuration or initramfs content issue. From the GRUB command line, booting manually:

grub> linux (hd0,gpt2)/boot/vmlinuz-6.8.0-1015-generic root=/dev/sda2 ro
grub> initrd (hd0,gpt2)/boot/initrd.img-6.8.0-1015-generic
grub> boot

This got the system far enough to reach a root shell — not a fully working boot, but enough to diagnose and fix the actual cause from inside the VM.

The actual root cause: missing modules.dep

Once at a shell, regenerating the initramfs surfaced the real problem:

update-initramfs -c -k 6.8.0-1015-generic
# W: missing /lib/modules/6.8.0-1015-generic/modules.dep!

The linux-generic package had installed, but depmod — which builds the module dependency map the initramfs generation relies on — hadn't run correctly. Doing the install-and-purge in the same session, without an intervening reboot, left dpkg's package configuration hooks in an inconsistent state where this dependency step silently didn't complete.

The fix:

depmod -a 6.8.0-1015-generic
update-initramfs -c -k 6.8.0-1015-generic
update-grub

-c (create) rather than -u (update) was the deliberate choice — the existing initramfs was the suspected malformed artifact, so regenerating from scratch was safer than patching it in place.

After this, a normal reboot from the Azure portal came up cleanly on linux-generic with SSH reachable again.

Why the reboot between install and purge matters

The general advice — reboot and confirm before purging the old kernel — is usually framed as "keep a fallback GRUB entry available." The depmod root cause adds a second, more specific reason: a reboot between installing the new kernel and removing the old one forces a clean package configuration state. The same-session install-then-purge sequence can skip past the depmod step that update-initramfs depends on, producing a malformed initramfs that only surfaces as a boot failure after the reboot you skipped.

Troubleshooting checklist for serial-console-reachable boot failures

In order of how often each turns out to be the actual cause:

Confirm kernel and initramfs files exist on disk — ls from the GRUB command line before assuming anything about why boot is failing
Check modules.dep and run depmod -a explicitly if initramfs generation logs mention missing module dependencies — a plain update-initramfs re-run won't fix it without depmod first
Inspect /etc/default/grub and /boot/grub/grub.cfg for a default entry pointing at a kernel version that no longer exists
Check disk space on /boot — a full /boot partition can silently truncate update-initramfs or update-grub without a clear error, only surfacing as a boot failure afterward
Consider a snapshot restore if manual recovery isn't converging — sometimes the right first move rather than the last resort, depending on time pressure

Pre-flight check for the next server in the wave

Two changes folded back into the migration runbook after this incident:

Never purge the old kernel in the same session as installing the new one — always reboot and confirm first
Confirm boot diagnostics is enabled and serial console access is usable — role permissions included — on every source VM before starting kernel-level work

az vm boot-diagnostics get-boot-log --name <vm-name> --resource-group <rg-name> \
  > /dev/null 2>&1 && echo "OK: boot diagnostics reachable" \
  || echo "WARNING: boot diagnostics not enabled or not reachable — enable before kernel changes"

Read the Full Article

The full article covers:

Why this boot loop failure is structurally different from the AWS-side ena driver issue in the previous article — same symptom ("server unreachable after kernel change"), completely different cause and recovery tool
The full GRUB command-line recovery walkthrough with exact commands
Why update-initramfs -c rather than -u was the right choice once at a shell
The decision to do manual recovery rather than snapshot restore — and when snapshot restore is actually the better first move
The pre-migration pre-flight script extended to confirm serial console access before kernel work begins

👉 Recovering an Azure VM Boot Loop with Serial Console, Mid-Migration — Full Article

Progressive Delivery with Argo Rollouts: Canary, Blue-Green, and AnalysisTemplates That Actually Gate

Alok Ranjan Daftuar — Mon, 05 Oct 2026 11:40:13 +0000

A standard Kubernetes Deployment rolling update has no concept of "check whether this is actually working before replacing more Pods." It replaces old Pods with new ones at the pace you configured, and if the new version is broken, it finds out the same way your users do — by serving it to more and more of them until someone notices.

Argo Rollouts replaces the Deployment resource with a Rollout — API-compatible in its Pod template, but with an explicit strategy for how traffic shifts and, critically, automated checks that can halt or reverse that shift before it reaches everyone. This post covers the two strategies that matter in practice and the AnalysisTemplate mechanism that makes the gating real rather than cosmetic.

Why a Deployment Isn't Enough

A Deployment's rolling update is governed by maxSurge and maxUnavailable — how many extra Pods can exist during the rollout, how many can be unavailable. Neither is a correctness check. A Deployment will happily roll a broken version to 100% of Pods because nothing in its model asks "is this new version actually healthy" — only "are enough Pods reporting ready," which a broken-but-still-passing-liveness-probe version satisfies just fine.

The two strategies have fundamentally different blast-radius profiles:

Canary                                    Blue-Green
  100% ─┐                                   100% ─┐ old (active)
        │                                         │
   10%  │ old          10% new                new │ 100% (preview,
        │              ▲                           │      no live traffic)
   50%  │ old      50% │ new                        │
        │              │ analysis gate               │ prePromotionAnalysis
    0%  │              │ new: 100%                    │ gate
        └──────────────┴──────────────►          cutover: 100% new,
        traffic shifts in steps,                  0% old, in one move
        gated between each step

Canary spreads risk across several small, gated exposures. Blue-green concentrates verification into one pre-cutover check, then moves all traffic at once.

Canary Strategy

Canary shifts traffic to the new version incrementally, in named steps, with a pause (manual or timed) between each. The key YAML shape:

strategy:
  canary:
    steps:
      - setWeight: 10
      - pause: { duration: 5m }
      - analysis:
          templates:
            - templateName: checkout-success-rate
          args:
            - name: service-name
              value: checkout-service-canary
      - setWeight: 50
      - pause: { duration: 5m }
      - analysis:
          templates:
            - templateName: checkout-success-rate
      - setWeight: 100

Each setWeight step shifts that percentage of traffic to the new ReplicaSet via whichever traffic-management integration is configured — an Ingress controller, a service mesh, or a load balancer controller. The analysis steps between weight increases are where this stops being "a slower rolling update" and becomes an actual gate: the rollout does not proceed to the next setWeight unless the referenced AnalysisTemplate reports success.

The traffic-router gap worth knowing: if the Ingress controller isn't one of Rollouts' supported traffic-router plugins, it silently falls back to a ReplicaSet-ratio approximation. A single canary Pod out of 10 total is not the same as a precise 10% of requests. Confirming which traffic-router plugin is actually active — not just assumed — is worth doing before trusting setWeight values as precise.

Blue-Green Strategy

Blue-green skips the incremental traffic shift and instead runs the full new version alongside the full old version, with a single cutover once the new version is verified:

strategy:
  blueGreen:
    activeService: checkout-service-active
    previewService: checkout-service-preview
    autoPromotionEnabled: false
    prePromotionAnalysis:
      templates:
        - templateName: checkout-smoke-test
      args:
        - name: service-name
          value: checkout-service-preview
    scaleDownDelaySeconds: 300

The new version scales up fully behind previewService — reachable for testing, not yet receiving production traffic. prePromotionAnalysis runs against the preview version before any cutover is possible. With autoPromotionEnabled: false, cutover requires an explicit kubectl argo rollouts promote even after analysis passes — a deliberate human checkpoint on top of the automated one.

scaleDownDelaySeconds keeps the old ReplicaSet running for 5 minutes after cutover so a rollback is a service-selector flip back, not a fresh Pod startup. Setting it too low defeats the rollback advantage — if Pods take 90 seconds to become traffic-ready and scaleDownDelaySeconds is 30, the old ReplicaSet is already gone by the time a problem surfaces.

AnalysisTemplates — What Actually Gates the Rollout

Both strategies are only as good as the analysis behind them. An AnalysisTemplate defines a metric query and a success condition; Rollouts polls it on an interval and compares the result against thresholds:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: checkout-success-rate
  namespace: checkout
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 1m
      count: 5
      successCondition: result[0] >= 0.95
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}", status!~"5.."}[1m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[1m]))

This queries Prometheus for the canary's success rate every minute, five times, and requires each sample to hold at or above 95%. Rollouts supports the same pattern against Datadog, New Relic, CloudWatch, and others — the shape of "query a metric, define a success condition, sample N times" is the transferable part.

The two failure modes worth planning for explicitly:

A successCondition that's too loose (result[0] >= 0.50, left over from testing) will pass a canary that's failing half its requests. The gate is present but not gating.
A count too low to be statistically meaningful on a low-traffic service — one or two failures in a handful of requests swing the ratio well outside a 95% threshold on pure noise, triggering rollbacks on releases that were actually fine.

Both point to the same discipline: AnalysisTemplate thresholds need to be tuned against the actual traffic volume and baseline error rate of the specific service, not copied from another service's template.

Where This Sits Against CI/CD Pipeline Stages

Kargo's Stage-level verification (or a CI-to-GitOps handoff) answers "should this artifact be allowed into this environment at all." Argo Rollouts answers a different question, one layer down: "given that this artifact is now allowed into this environment, how carefully should it be exposed to that environment's traffic."

The two compose rather than compete. A Kargo Stage promoting into prod can just as easily be updating a Rollout resource as a plain Deployment. Nothing about progressive delivery requires picking a different CI-to-GitOps mechanism.

Running both isn't redundant defense in depth for its own sake — each catches a class of failure the other structurally cannot. A service can pass every pre-promotion check and still fail an in-flight Rollouts analysis: a load-dependent memory leak, a connection pool exhausting under real concurrency, a downstream dependency behaving differently under production traffic patterns than under a synthetic staging test.

Decision Framework

Situation	Recommended strategy
Stateless service, traffic-splitting infrastructure already in place	Canary — gradual blast-radius limiting, most granular control
No traffic-splitting infrastructure available	Blue-green, or canary via ReplicaSet ratio approximation
Release needs real-traffic-pattern validation before any production exposure	Blue-green — preview service gives you that validation window
Workload has strict version-coupling to a shared schema	Neither — fix the deployment pattern first

Read the Full Article

The full post covers:

Complete annotated YAML for both canary and blue-green Rollout resources
The exact traffic-router gap that makes setWeight values imprecise without a supported plugin
Why scaleDownDelaySeconds sizing directly determines whether blue-green's rollback advantage is real or theoretical
How AnalysisTemplate count and successCondition interact with low-traffic services to produce false rollbacks
The precise relationship between Kargo/CI promotion gates and in-flight Rollouts analysis — why both are required and what each catches that the other cannot

👉 Progressive Delivery with Argo Rollouts — Full Article

The MCP Stateless Revolution

Alok Ranjan Daftuar — Thu, 01 Oct 2026 10:58:38 +0000

For two years, running an MCP server in production meant building infrastructure that had nothing to do with what the server was actually supposed to do. The protocol's session model forced sticky routing, shared session stores, and affinity cookies onto teams who otherwise had no reason to build stateful infrastructure at all. On July 28, 2026, the MCP specification removed the session entirely — and with it, the justification for that entire category of workaround.

What the old transport forced on you

An MCP initialize handshake returned an Mcp-Session-Id header, and every subsequent request in that conversation had to carry the same ID back to the same server instance. Keeping that session alive in production meant one of three things: sticky sessions at the load balancer, a shared session store like Redis so any instance could serve any request, or a client-side retry strategy that tolerated session loss and re-initialized when a pod restarted.

Each solution adds a moving part that fails in a specific, familiar way during a rolling deployment. A pod gets terminated, its in-memory session state goes with it, and every client pinned to that pod either gets a connection reset mid-conversation or silently starts talking to a session that no longer exists.

# Pre-2026-07-28: sticky routing required
apiVersion: v1
kind: Service
metadata:
  name: mcp-server
  annotations:
    alb.ingress.kubernetes.io/target-group-attributes: |
      stickiness.enabled=true,
      stickiness.type=lb_cookie,
      stickiness.lb_cookie.duration_seconds=3600

What SEP-2567 actually removes

The core change removes protocol-level sessions and the Mcp-Session-Id header from the Streamable HTTP transport entirely — not deprecates, removes. Every request is now self-describing: protocol version, client identity, and capabilities travel in _meta on that request rather than being negotiated once and remembered by a pinned server instance.

This does not mean MCP servers can no longer carry state across calls. If a server needs to remember something between tool calls, it mints an explicit handle and returns it from the first call. The client passes that handle back as an ordinary argument — the same way any REST API has always handled state. What disappears is the requirement that the same pod be the one to look it up.

Header-based routing: the other half of the change

Statelessness solves scaling, but creates a new problem: how does a gateway make intelligent routing decisions without parsing every JSON-RPC body? SEP-2243 answers this: Streamable HTTP requests must now carry Mcp-Method and Mcp-Name headers. A load balancer can now route, throttle, and meter MCP traffic on headers — without ever being MCP-aware at the body level.

# AWS ALB post-2026-07-28 — no stickiness, header-based routing
annotations:
  alb.ingress.kubernetes.io/conditions.mcp-tasks: |
    [{"field":"http-header","httpHeaderConfig":{"httpHeaderName":"Mcp-Method","values":["tasks/*"]}}]
  alb.ingress.kubernetes.io/actions.mcp-tasks: |
    {"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"mcp-tasks-pool","servicePort":8443}]}}

Long-running task calls route to a differently-sized backend pool. Ordinary tool calls go elsewhere. No custom MCP-parsing logic in the gateway.

Per-request auth: the security implication

Removing the session also removes the one place authentication used to be established once and implicitly trusted for the rest of the conversation. Every request now carries its own client identity in _meta, which means authorization is evaluated per request. A revoked credential takes effect on the very next call instead of only once the session naturally expires. The infrastructure implication: your gateway now performs an identity check on every request rather than once per conversation — real added latency on every call, worth budgeting for explicitly when sizing the gateway layer.

The migration window

The specification's deprecation policy gives a twelve-month compatibility runway for features being phased out. What is removed outright — with no grace period — is the session header and initialize handshake. The realistic migration path is running both transport paths side by side behind the same ingress until every client you support has moved. The sticky-routing configuration doesn't disappear the day you upgrade — it stays live serving the old path while new traffic routes to the stateless pool through header-based rules.

Read the Full Article

This summary covers the core protocol changes and their infrastructure implications. The full article includes:

Complete before/after Kubernetes Service and Ingress YAML for both AWS ALB and Azure Application Gateway
The ttlMs/cacheScope caching signals (SEP-2549) and what they mean for CDN/edge caching
W3C Trace Context propagation in _meta (SEP-414) and distributed tracing implications
How the removal of Last-Event-ID stream resumability simplifies liveness probe design
Detailed dual-path migration strategy to avoid mid-migration outages

👉 The MCP Stateless Revolution — Full Article

VPC Endpoints: Gateway vs Interface, and the AWS Traffic That Shouldn't Touch the Internet

Alok Ranjan Daftuar — Mon, 28 Sep 2026 05:31:14 +0000

Traffic from a private subnet to S3 routes through your NAT gateway by default — and gets billed per GB for the privilege — when a free alternative has existed for years. VPC endpoints give AWS services a private, in-VPC address path that never touches the internet. There are two fundamentally different mechanisms, and understanding how they differ changes how you decide which services actually need one.

Why private subnet traffic hits NAT at all

A private subnet instance calling s3.ap-south-1.amazonaws.com does exactly what it would do calling any internet host: routes to the NAT gateway, gets source-translated, exits through the IGW, and comes back the same way. Both the instance and the S3 bucket are inside AWS — the traffic never needed to leave AWS's network. VPC endpoints fix this by giving the traffic a direct in-VPC path.

Gateway endpoints: a route table entry, nothing more

Gateway endpoints exist for exactly two services: S3 and DynamoDB. They are not network devices — they are route table targets. AWS injects a managed prefix list route automatically; traffic matching S3's IP ranges routes to the endpoint instead of NAT.

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.ap-south-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]
}

Cost: free. No hourly charge, no per-GB processing charge. This makes the S3 Gateway endpoint close to a strict upgrade — there is essentially no reason not to add it to every VPC with private subnets that touch S3.

The one real limitation: Gateway endpoints are associated with specific route tables, not the whole VPC. A new private subnet with its own route table falls back to NAT silently unless you explicitly associate the endpoint with that table too. Worth adding to any subnet-creation checklist.

Interface endpoints: a real ENI, backed by PrivateLink

Every other AWS service that supports VPC endpoints uses an Interface endpoint — a real Elastic Network Interface provisioned in a subnet you specify, with a private IP from that subnet's range, backed by AWS PrivateLink.

resource "aws_vpc_endpoint" "secretsmanager" {
  vpc_id             = aws_vpc.main.id
  service_name       = "com.amazonaws.ap-south-1.secretsmanager"
  vpc_endpoint_type  = "Interface"
  subnet_ids         = [aws_subnet.private_a.id, aws_subnet.private_b.id]
  security_group_ids = [aws_security_group.vpce_sm.id]
  private_dns_enabled = true
}

Because it's a real ENI, it has its own security group — Interface endpoint access is controlled by security group rules, not route table association. private_dns_enabled = true makes the endpoint transparent to application code: AWS overrides the public DNS name to resolve to the endpoint's private IP inside the VPC, so no SDK or application configuration change is needed.

Cost: the opposite of Gateway endpoints. Interface endpoints bill hourly per AZ plus per-GB data processed. "Add Interface endpoints for everything" is not automatically correct — it requires actual usage data.

Which services actually justify an Interface endpoint

Start with services every private instance calls constantly regardless of application logic:

STS — IAM role assumption is called far more often than people expect under the hood
CloudWatch Logs — continuous log shipping from busy applications
ECR — image pulls from private subnets, especially during autoscaling cold starts or frequent CI/CD deployments

These three tend to justify their hourly cost quickly in any account with meaningful private-subnet compute. Lower-traffic services — Secrets Manager called once at startup, SNS for occasional notifications — are worth evaluating with real Cost Explorer data rather than provisioning preemptively.

aws ce get-cost-and-usage \
  --time-period Start=2026-07-01,End=2026-08-01 \
  --granularity MONTHLY \
  --filter file://nat-gateway-filter.json \
  --metrics "UsageQuantity" "BlendedCost"

Side-by-side comparison

	Gateway Endpoint	Interface Endpoint
Mechanism	Route table entry	ENI backed by PrivateLink
Services	S3, DynamoDB only	Most other AWS services
Cost	Free	Hourly per AZ + per-GB processed
Security control	Existing SG/NACL on the instance	Own security group on the endpoint ENI
DNS	No change needed	Private DNS override, if enabled
Multi-subnet	Associate route table per subnet	Deploy ENI per AZ needed

Failure modes worth knowing before you're debugging under pressure

Private DNS conflicts in Transit Gateway / peered VPC designs — if two connected VPCs both have Interface endpoints for the same service with private DNS enabled, DNS resolution can become ambiguous. An instance in VPC-A may resolve the service name to VPC-B's endpoint ENI. Test this explicitly in hub-and-spoke designs.
Security group too narrow — a security group that misses the calling subnet's CIDR produces a connection timeout that looks identical to a routing failure, but is actually a security group issue at the endpoint itself.
Endpoint deployed in wrong AZ — the endpoint still functions cross-AZ, but incurs cross-AZ data transfer charges on top of the endpoint's per-GB cost. Deploy into every AZ that has calling instances.
Custom DHCP option set or Route 53 Resolver rule taking precedence — if the VPC has custom DNS resolution configured, the endpoint's private hosted zone override can be superseded and traffic reverts to NAT despite correct endpoint configuration.

Read the Full Article

This summary covers the core mechanisms and decision framework. The full article includes:

Complete Terraform for both Gateway and Interface endpoints with security group scoping
The full AWS CLI route table inspection workflow
Detailed cost modelling for Interface endpoint break-even analysis
Security group and NACL implications in depth
The specific action to take this week if you haven't added the S3 Gateway endpoint yet

👉 VPC Endpoints: Gateway vs Interface — Full Article

Secrets Management in a GitOps World: Sealed Secrets vs. External Secrets Operator vs. Vault

Alok Ranjan Daftuar — Thu, 24 Sep 2026 09:03:22 +0000

GitOps means everything a cluster needs lives in Git. Secrets very much don't belong in Git — not even in a private repo, because Git history is forever and "who had read access eighteen months ago" is a much harder question than "who has read access today."

Three approaches resolve that contradiction. Each breaks the "it's all just committed to Git" assumption in a different place, and each solves a different subset of the actual problem.

What "Solving" Secrets in GitOps Actually Means

Before comparing tools, it's worth being precise about what's being solved — because the three approaches solve different parts of it:

Never having plaintext in Git — not even encrypted-and-committed, if the encryption key itself can leak
Rotating secrets without a Git commit — if rotation requires a PR, it happens less often than it should
Scoping which cluster or namespace can see which secret — the same multi-tenancy question AppProjects answer for Applications, but for secret values

No single approach scores well on all three by default. Each is a different set of tradeoffs.

Sealed Secrets

Bitnami's Sealed Secrets takes the most literal path: encrypt the secret client-side with a public key, commit the resulting SealedSecret resource, and let an in-cluster controller — the only holder of the matching private key — decrypt it at apply time.

kubectl create secret generic checkout-db-creds \
  --dry-run=client \
  --from-literal=password='REPLACE_ME' \
  -o yaml | kubeseal --format yaml > checkout-db-creds-sealed.yaml

# safe to commit — only the in-cluster controller can decrypt this
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: checkout-db-creds
  namespace: checkout
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEQ...

What it gets you: nothing sensitive ever touches Git. No external dependency — self-contained controller, no Vault cluster or cloud secrets service required.

What it doesn't get you: rotation still requires a commit. Changing a database password means re-sealing and re-committing — same PR friction as any other manifest change. At 40 services rotating every 90 days, that's 40 separate re-seal-and-commit operations per quarter, each needing someone with kubeseal access to the right cluster's public key at the time rotation is due.

It's also cluster-scoped by design: a SealedSecret sealed for cluster A's public key is meaningless on cluster B. At multi-cluster scale, that means either re-sealing per cluster or restricting Sealed Secrets to genuinely cluster-local secrets.

External Secrets Operator

ESO inverts the model entirely: the secret's actual value never enters Git at all. What's committed is a reference — "fetch this key from AWS Secrets Manager" — and a controller resolves that reference into a real Secret by calling the secrets backend at sync time.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: checkout-db-creds
  namespace: checkout
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: checkout-db-creds
  data:
    - secretKey: password
      remoteRef:
        key: checkout-service/prod/db-password

This manifest is entirely safe to commit — there's no ciphertext, no encoded value. The sensitive part is the value at checkout-service/prod/db-password, which lives only in AWS Secrets Manager.

What it gets you: rotation with zero Git commits — update the value in Secrets Manager and every ExternalSecret referencing it picks up the change on its next refreshInterval cycle. Centralized secret storage that extends your existing cloud secrets service into Kubernetes rather than maintaining a separate story.

What it doesn't get you: the secrets backend must be reachable from every cluster running ESO — reintroducing the network-reachability planning from multi-cluster architecture, this time for every spoke's path to Secrets Manager or Key Vault.

One scoping risk worth calling out: a SecretStore backed by an overly broad IAM role — one that can read every key under checkout-service/* rather than the specific key a given ExternalSecret needs — means a misconfigured ExternalSecret in one namespace can pull secrets it was never meant to see. Scope the IAM policy behind each SecretStore to the narrowest key path prefix that namespace actually owns.

Vault Agent Injector

Where ESO pulls a secret into a Kubernetes Secret object, Vault's Kubernetes integrations more often avoid creating a Secret object at all. The Vault Agent Injector mounts secrets directly into a Pod's filesystem as files, at Pod startup, authenticated via Kubernetes' own service account token exchanged against Vault's Kubernetes auth method.

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: checkout-service
        vault.hashicorp.com/agent-inject-secret-db-creds: secret/data/checkout-service/prod/db
    spec:
      serviceAccountName: checkout-service
      containers:
        - name: checkout-service
          # application reads from /vault/secrets/db-creds — never from an env var

What it gets you: the closest thing to "the secret was never a Kubernetes API object" — no Secret resource exists to be over-broadly RBAC'd, listed via kubectl get secrets, or accidentally dumped in a debug script.

More importantly: Vault's database secrets engine can generate a unique, short-lived database username and password per Pod, on demand, with a lease duration Vault itself enforces. A 1-hour lease means Vault revokes that specific credential at the database level after an hour, independent of whether anyone remembered to rotate anything. This changes rotation from "a scheduled human task" into "an enforced property of every credential issued" — a meaningfully different compliance posture.

What it doesn't get you: simplicity. Running Vault well — unsealing, storage backend, HA, policy management — is a larger operational commitment than either of the other two approaches. If Vault isn't already part of your platform, adopting it purely to solve the GitOps secrets problem is usually the wrong-sized tool.

Decision Framework

Situation	Recommended approach
Small team, few clusters, low rotation cadence	Sealed Secrets — simplest, no external dependency
Already using AWS Secrets Manager or Azure Key Vault	External Secrets Operator — extends existing storage into Kubernetes
Multi-cluster fleet, secrets needed consistently across clusters	External Secrets Operator — reference model isn't cluster-scoped
Already running Vault for dynamic credentials or PKI	Vault Agent Injector — extends a tool you're already operating
Secrets where even a `Secret` object existing in the cluster is too much exposure	Vault injector — no `Secret` object is ever created

These aren't exclusive. Running ESO for the bulk of a fleet's secrets and reserving Vault's injector for the small set of credentials needing dynamic short-lived leases is a reasonable production posture — matching each secret's sensitivity to the approach that handles it well.

Read the Full Article

The full article covers:

The precise three-part definition of what "solving" secrets in GitOps actually means — and why no single tool covers all three
The 90-day rotation cost at 40 services with Sealed Secrets, and why it surfaces at compliance audit time
The cluster-scoping limitation of Sealed Secrets at multi-cluster scale
Full SecretStore + ExternalSecret YAML with IRSA/workload-identity auth chain
The IAM scoping risk with ESO and how it mirrors AppProject destinations restriction
Vault's dynamic credential model explained — why it changes rotation from a task to an enforced property
How all three approaches compose in a real fleet rather than requiring a single tool choice

👉 Secrets Management in a GitOps World — Full Article