DEV Community: Petr Petrenko

5 Istio misconfigurations that istioctl analyze won't catch

Petr Petrenko — Wed, 19 Aug 2026 07:12:01 +0000

istioctl analyze is the go-to tool for validating Istio configuration. It catches unknown hosts, missing Services, malformed specs. Good tool.

But there is a class of problem it does not catch: configurations that parse correctly but break traffic at runtime. No schema violation, no warning, just silent failures or hard-to-trace 503s.

Here are five I keep running into.

1. No mTLS enforcement

A namespace with no PeerAuthentication runs in PERMISSIVE mode by default: Istio accepts both plaintext and mTLS traffic. Your sidecar can encrypt, but nothing forces it to.

istioctl analyze is fine with this. No PeerAuthentication at all is a valid configuration.

The problem shows up when you assume mTLS is on and it isn't. For example, when a new namespace gets created and nobody adds the policy.

What you need:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT

Without this, plaintext traffic enters your namespace silently.

2. VirtualService with retries but no timeout

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: payment-service
spec:
  http:
  - retries:
      attempts: 3
      perTryTimeout: 5s
    route:
    - destination:
        host: payment-service

This looks reasonable: three retry attempts, five seconds each. But there is no top-level timeout. If the upstream is slow or stuck, each attempt waits the full 5s and all three fire. One slow request from the client turns into up to 15 seconds of upstream load.

In a chain of services, this compounds. Three hops with the same config and a stuck dependency: 45 seconds of retry storm, all traced back to one request.

What you need:

    timeout: 10s
    retries:
      attempts: 3
      perTryTimeout: 3s

istioctl analyze has no opinion on retry/timeout combinations.

3. DestinationRule with no outlier detection

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: order-service
spec:
  host: order-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100

This DestinationRule is valid. But without outlier detection, there is no automatic ejection of pods that are degraded but still running.

Kubernetes readiness probes handle the obvious case: a pod that crashes or fails its health check gets removed from Endpoints. But a pod that is alive and passing its health check while returning 500s or responding very slowly stays in rotation. Outlier detection catches exactly this: it watches observed error rates, not just liveness.

Add this:

    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s

istioctl analyze has no opinion on missing outlier detection.

4. Running pods without an Istio sidecar

A pod without istio-proxy is outside the mesh. No mTLS, no telemetry. This can happen when:

A namespace has injection enabled but a specific Deployment has sidecar.istio.io/inject: "false"
A pod was created before injection was enabled and was never restarted
An operator or Job runs with injection disabled and nobody noticed

istioctl analyze won't flag this. The pod is running, the config is valid.

This is especially painful because the pod will still receive traffic, but it won't be mTLS-encrypted, and VirtualService routing rules won't apply to outbound traffic it initiates.

5. EnvoyFilter without a workloadSelector

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: add-header
  namespace: production
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    ...

No workloadSelector. This EnvoyFilter applies to every pod in the namespace. If it is in istio-system, it applies to every pod in the entire mesh.

EnvoyFilters are the most powerful and most dangerous Istio primitive. A misconfigured patch pushed without a selector has taken down whole clusters.

istioctl analyze validates the patch structure, not the blast radius.

Catching these automatically

I built meshscan to catch exactly this class of problem. It reads your cluster's Istio resources and reports semantic misconfigurations: things that are syntactically valid but operationally dangerous.

$ meshscan -n production

meshscan report: namespace production
4 issues: 1 critical, 2 high, 1 medium

[CRITICAL] PeerAuthentication
  check: mtls-enforcement
  no PeerAuthentication found; namespace defaults to PERMISSIVE

[HIGH] DestinationRule/order-service
  check: outlier-detection
  no outlier detection configured

[HIGH] VirtualService/payment-service
  check: retry-without-timeout
  retries.attempts=3 but no timeout set (unbounded retry amplification possible)

[MEDIUM] VirtualService/order-service
  check: missing-dr
  routes to order-service but no DestinationRule exists

It scans a single namespace or the whole cluster with -A. JSON output for CI pipelines. No cluster-side install needed: just a kubeconfig and list RBAC permissions.

go install github.com/n0rm4l-me/meshscan/cmd/meshscan@latest

The two tools complement each other. Run istioctl analyze for schema and config validation. Run meshscan for runtime semantic issues.

Have you hit any of these in production? Curious what other silent failure modes people have run into.

Contributing a GCP Cloud Spanner Scaler to KEDA

Petr Petrenko — Wed, 05 Aug 2026 01:17:45 +0000

How we added native Spanner support to Kubernetes autoscaling — and what we learned along the way

The problem

We run several workloads on Kubernetes that process jobs stored in Cloud Spanner tables. The pattern is simple: a producer writes rows with status = 'pending', workers pick them up and mark them done. The question is — how many workers do you run?

Fixed replica counts mean either wasted money during quiet periods or dropped throughput during spikes. We needed autoscaling based on actual queue depth, not CPU or memory.

KEDA (Kubernetes Event-Driven Autoscaling) is the standard answer for this — it scales workloads based on external metrics like queue lengths, database counts, and custom queries. It already had scalers for GCP Pub/Sub, Cloud Tasks, and Cloud Storage. But not Spanner.

So we built one.

How KEDA works

KEDA sits between your workload and the external system. On every polling interval it runs your query, gets a number back, and tells Kubernetes HPA how many replicas to run based on ceil(currentValue / targetValue).

The scaler

The gcp-spanner trigger takes any SQL query that returns a single INT64 value:

triggers:
  - type: gcp-spanner
    metadata:
      projectId: my-project
      instanceId: my-instance
      databaseId: my-database
      query: "SELECT COUNT(*) FROM jobs WHERE status = 'pending'"
      targetValue: "5"       # one replica handles 5 pending jobs
      activationValue: "2"   # stay at 0 replicas below this threshold
      credentialsFromEnv: GOOGLE_APPLICATION_CREDENTIALS_JSON

With targetValue: 5 and 20 pending jobs, KEDA will maintain 4 worker replicas. When the queue drains to 0, it scales back to zero.

Scaling behaviour

What we learned contributing to KEDA

1. Schema generation matters

KEDA auto-generates its scaler schema from Go struct tags. We needed to add Credentials and CredentialsFromEnvFile fields to the metadata struct purely so they appear in the schema — even though GetGCPAuthorization reads them directly from the config. This is the same pattern used by other GCP scalers.

2. ParseCommand has a quirk

KEDA's e2e test helper splits commands on spaces, honouring single-quotes only. So --ddl="value with spaces" breaks — "value" doesn't get treated as a quoted string. The fix is --ddl 'value with spaces' — space-separated flag and value, with single quotes.

3. Cleanup must be independent of Kubernetes

Our first attempt deleted the Spanner test instance via gcloud running in a pod. But DeleteKubernetesResources removes the pod before t.Cleanup fires — leaving orphaned (and billing) Spanner instances. The fix: delete via the Go Spanner Admin API directly from the test binary, completely independent of any pod.

4. targetValue: 0 causes HPA divide-by-zero

We added a Validate() method to reject targetValue <= 0 — caught by a Copilot review comment. KEDA's TypedConfig calls Validate() automatically via the CustomValidator interface.

The PR

The contribution includes:

Scaler implementation with all three GCP auth methods (inline JSON, env file, Workload Identity)
18 unit tests
6 integration tests against the Cloud Spanner emulator
e2e test that provisions a real Spanner instance, runs scaling scenarios, and cleans up

PR: kedacore/keda#7844

Docs: keda.sh — GCP Spanner scaler

Using it

kubectl apply -f - <<EOF
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: spanner-worker-scaler
spec:
  scaleTargetRef:
    name: job-processor
  minReplicaCount: 0
  maxReplicaCount: 20
  triggers:
    - type: gcp-spanner
      metadata:
        projectId: my-project
        instanceId: my-instance
        databaseId: my-database
        query: "SELECT COUNT(*) FROM jobs WHERE status = 'pending'"
        targetValue: "5"
        activationValue: "2"
        credentialsFromEnv: GOOGLE_APPLICATION_CREDENTIALS_JSON
EOF

That's it. KEDA will handle the rest.

Your Database Just Died. Why Is Everything Still Running?

Petr Petrenko — Mon, 15 Jun 2026 05:50:46 +0000

It's 3 AM. Your PostgreSQL pod crashes. On-call fires. The engineer wakes up, checks the dashboard, and spends 20 minutes figuring out why the payments service is throwing 500s at 100% error rate before they realize — database is down, but payments is still running, hammering connection pools, logging thousands of errors per second, triggering cascading alerts.

They scale payments to zero. Problem stops. Recovery begins.

That 20 minutes was avoidable. klink would have scaled payments to zero automatically — 30 seconds after the database went down.

The Problem Nobody Talks About

Kubernetes is excellent at keeping individual services running. Liveness probes, restart policies, resource limits — it's all there. But Kubernetes has no concept of relationships between workloads.

When your database fails, Kubernetes does exactly what you told it to: keeps the dependent services running. Those services now do nothing useful — they just generate noise, consume resources, and make your incident harder to debug.

This is the gap klink fills.

What klink Does

klink introduces a new primitive: WorkloadDependency. You declare that service B depends on service A. klink watches. When A goes unhealthy, klink scales B to zero. When A recovers, klink restores B automatically.

apiVersion: deps.klink.dev/v1alpha1
kind: WorkloadDependency
metadata:
  name: payments-needs-database
  namespace: production
spec:
  dependent:
    kind: Deployment
    name: payments-service

  dependsOn:
    - kind: Deployment
      name: postgresql
      condition:
        minReadyPercent: 80    # healthy if ≥80% pods ready
        window: 30s            # ignore transient restarts
        recoveryWindow: 60s    # wait for stability before restoring

  onDegraded:
    action: ScaleToZero

  mode: strict

That's it. No code changes. No sidecars. No complex configuration.

How It Works

The hysteresis window is critical. Without it, a single pod restart would cascade a shutdown. With window: 30s, klink ignores transient failures — only sustained outages trigger action.

Enforcement Modes

Different situations call for different behavior. klink has four modes:

Mode	What it does	When to use
`strict`	Scales to 0 on failure. Reverts manual scale-ups within 15s.	Production services where cascade is non-negotiable
`soft`	Scales to 0 once. Respects manual overrides.	Services where operators need flexibility during incidents
`gate`	Doesn't scale down. Blocks scale-up via admission webhook.	Preventing HPA from scaling up while dependency is down
`observe`	Logs what it would do. Takes no action.	Safe onboarding — see what klink would do before enabling it

Start with observe mode. Apply klink to your existing services, watch the logs for a week, and only switch to strict or soft once you're confident.

spec:
  mode: observe  # "would scale payments to 0 — dependency unhealthy"

The Mutual Dependency Problem

What happens when A depends on B and B depends on A?

Naive implementations deadlock. Both services go to zero, each waiting for the other to recover. You need a manual fix every time.

klink solves this with CoSuspended detection:

When klink scales payments to zero because database failed, it marks payments as CoSuspended — intentionally scaled down by klink, not actually broken.

When database checks its dependencies, it sees payments at zero — but recognizes it as CoSuspended and doesn't cascade. When you manually restore database, klink automatically restores payments.

No deadlock. No manual intervention.

Argo Rollout Support — Canary Awareness

klink understands Argo Rollouts. If your payments service is in the middle of a canary deployment when its database goes down, klink defers the scale-to-zero until the rollout completes.

You never want to interrupt an active deployment. klink handles this automatically.

CronJob Support — Suspend Instead of Scale

For batch jobs, scaling to zero makes no sense. klink sets spec.suspend: true instead:

spec:
  dependent:
    kind: CronJob
    name: nightly-billing-export
  dependsOn:
    - kind: Deployment
      name: billing-service

When billing-service goes down, the CronJob is suspended. No failed jobs accumulating in history. When billing-service recovers, the CronJob resumes automatically.

Notifications

Get notified when workloads are suspended or restored — before your monitoring fires:

spec:
  notify:
    webhookSecretRef:
      name: slack-webhook
      key: url
    onPhases: [Suspended, Healthy]

The notification arrives the moment klink acts, with full context:

{
  "workloadDependency": "payments-needs-database",
  "namespace": "production",
  "phase": "Suspended",
  "previousPhase": "Degraded",
  "dependent": "payments-service",
  "dependentKind": "Deployment",
  "message": "dependency postgresql not healthy: 0/3 ready",
  "timestamp": "2026-06-15T03:00:00Z"
}

Notifications include retry with exponential backoff (1s → 2s → 4s) so transient webhook outages don't silently drop alerts.

Safety Net — maxSuspendDuration

Long outages happen. Your database might be down for hours. You don't want your payments service suspended indefinitely.

spec:
  onDegraded:
    action: ScaleToZero
    maxSuspendDuration: 4h

After 4 hours, klink restores the workload regardless of dependency state and enters Released phase — it won't re-suspend until the dependency genuinely recovers. This prevents indefinite outages from a single bad dependency.

Observability

klink exports Prometheus metrics so you can see exactly what's happening:

klink_dependency_phase{namespace="production", name="payments-needs-database", phase="Suspended"} 1
klink_scale_to_zero_total{namespace="production", kind="Deployment", name="payments-service"} 3
klink_replicas_restored_total{namespace="production", kind="Deployment", name="payments-service"} 3

GKE users get a PodMonitoring resource automatically when metrics are enabled.

Getting Started

helm upgrade --install klink oci://ghcr.io/n0rm4l-me/charts/klink \
  --version 0.3.0 \
  --namespace klink-system \
  --create-namespace

Apply your first WorkloadDependency:

apiVersion: deps.klink.dev/v1alpha1
kind: WorkloadDependency
metadata:
  name: payments-needs-database
  namespace: default
spec:
  dependent:
    kind: Deployment
    name: payments
  dependsOn:
    - kind: Deployment
      name: database
      condition:
        minReadyPercent: 80
        window: 30s
        recoveryWindow: 60s
  onDegraded:
    action: ScaleToZero
  mode: observe  # start here — see what klink would do

Check the status:

kubectl get workloaddependencies -A

NAMESPACE    NAME                      PHASE     REPLICAS   MESSAGE
production   payments-needs-database   Healthy              all dependencies healthy

When you're comfortable with what you see in observe mode, switch to strict or soft.

What klink Supports

Workload	As dependent	As dependency
Deployment	✅ scale to 0	✅ readyReplicas check
StatefulSet	✅ scale to 0	✅ readyReplicas check
CronJob	✅ suspend/resume	—
Argo Rollout	✅ scale to 0 (canary-aware)	✅ phase check

The Incident That Started This

We run microservices on Kubernetes. One evening our message queue had a rolling restart — routine maintenance, 45 seconds of unavailability. But 12 services that depended on it kept running and kept trying to connect. By the time the queue was back, we had retries queued up, connection pools exhausted, and a 10-minute degraded period that should have been 45 seconds.

The fix was conceptually simple: "if the queue is down, pause the consumers." But there was no Kubernetes-native way to express that relationship.

So we built klink.

What's Next

Prometheus-based health conditions — promQuery: 'pg_up == 1' instead of readyReplicas
kubectl klink plugin — klink graph, klink why payments-service
DaemonSet support

The project is open source under Apache 2.0. Issues, PRs, and feedback welcome.

GitHub: github.com/n0rm4l-me/klink

We replaced etcd with Google Cloud Spanner. Here's what happened.

Petr Petrenko — Tue, 09 Jun 2026 04:08:55 +0000

spanner-etcd is an open source (Apache 2.0), drop-in etcd v3 replacement backed by Google Cloud Spanner. Same API, no client changes — just point --etcd-servers at it.

We built it because etcd has a fundamental scaling constraint: every write serializes through a single global revision counter. One row, one lock, every transaction waits in line. At 32 concurrent writers, that counter becomes the bottleneck.

The GKE team solved this years ago internally to scale Kubernetes to 65,000 nodes. Their implementation is closed. So we built an open one.

This is the story of how it works, what we got wrong, and the honest benchmark numbers.

The core idea: timestamps as revisions

etcd's revision is a monotonically increasing integer. Every write increments it. That increment is the serialization point.

Spanner has PENDING_COMMIT_TIMESTAMP() — a TrueTime-based timestamp assigned at commit time, globally unique, strictly monotonic across all transactions. No counter. No lock. Each transaction commits independently.

So instead of:

UPDATE kv_rev SET rev = rev + 1 WHERE id = 1;  -- everyone waits here
INSERT INTO kv (rev, key, value) VALUES (42, '/foo', 'bar');

We do:

INSERT INTO kv (rev, key, value)
VALUES (PENDING_COMMIT_TIMESTAMP(), '/foo', 'bar');

The revision is the commit timestamp, cast to int64 UnixNano. Valid etcd ModRevision. Zero contention.

Result: at ×32 concurrency, write throughput went from a serialized bottleneck to 673 ops/sec — 15× faster than the integer counter baseline.

Watch events via Change Streams

etcd Watch is a streaming API — clients subscribe to a prefix and receive events as writes happen. In a vanilla etcd replacement you'd poll. We tried that first: it worked, but ~1s latency felt wrong.

Spanner has Change Streams — a push-based CDC mechanism that delivers row changes within tens of milliseconds. We built a partition reader that:

Starts streaming all partitions of kv_changes
Persists partition cursors to Spanner every 5s (so replicas resume correctly after restart)
Falls back to 1-second polling on the emulator (Change Streams aren't supported there)

The result: ~30ms Watch latency end-to-end on production Spanner in the same region. Not etcd's 1ms — Spanner is not a local in-memory store. But for Kubernetes workloads it's completely fine.

The back-join problem we almost missed

Early benchmarks showed Get at 71 ops/sec. Seemed reasonable. Then we looked at the query plan.

Our schema used PRIMARY KEY (id) with a bit_reversed_positive sequence — standard Spanner advice to avoid write hotspots. The secondary index kv_key_rev ON kv(key, rev DESC) existed for reads. But Spanner was doing this for every Get:

Index scan kv_key_rev → find the row's id
Table lookup on kv by id → fetch the actual data

Two round-trips inside one query. The fix was a single DDL change:

CREATE INDEX kv_key_rev ON kv (key, rev DESC)
  STORING (value, old_value, lease_id, deleted, created,
           create_revision, prev_revision);

STORING copies all needed columns into the index. Spanner can now serve reads entirely from the index — no back-join.

Get improved +40%. Mixed workload improved +167%. Measured before and after on the same hardware.

We also added kv_rev_desc ON kv(rev DESC) so CurrentRevision() does an O(1) LIMIT 1 seek instead of a full MAX(rev) scan.

One caveat: STORING value where value is BYTES(MAX) doubles write amplification for large values. For Kubernetes workloads (mostly small JSON/protobuf objects) this is fine. For blob storage it would be a problem.

Stateless replicas

This is the part that feels almost too simple. Because all state lives in Spanner, every replica is completely stateless. No consensus. No leader election between replicas. No split-brain.

We tested this explicitly: Watch on replica 2, writes through replica 1, then killed replica 1. Replica 2 received all events — before and after the kill — with zero gaps. 45 Watch streams migrated in ~10s. Kubernetes didn't notice.

The only statefulness is the Change Stream cursor, persisted to Spanner itself and recovered on restart. No leader election, no quorum, no split-brain scenario possible.

Real numbers

Everything below is production Spanner (regional-us-central1, 1000 PU), same-region e2-standard-4 VM, not the emulator.

Throughput:

Operation	ops/sec	Latency
Create ×1	90	11.1ms
Create ×4 parallel	270	3.7ms
Get ×1	108	9.3ms
Get ×4 parallel	481	2.1ms
Mixed ×4 (70% read)	403	2.5ms
Watch latency	—	~30ms

How many Spanner PUs do you actually need?

We benchmarked at 100, 1000, and 2000 PU on us-central1:

Operation	100 PU	1000 PU	2000 PU
Create ×4 parallel	87	270	255
Get ×4 parallel	472	481	469
Mixed ×4	294	403	404
Watch latency	29ms	~30ms	30ms

Interesting findings:

Single-key ops are nearly identical across 1000 and 2000 PU — you're paying for network round-trip, not Spanner compute (CPU was ~1% during benchmarks)
Parallel writes fall off sharply at 100 PU — Create ×4 drops from 270 to 87 ops/sec
Watch latency is consistent at ~30ms across all tiers
100 PU is enough for small clusters (< 100 nodes with moderate write rates)

Multi-region: what does global durability actually cost?

We ran one more experiment. We switched to nam6 — Iowa + South Carolina + Oregon + Los Angeles — and benchmarked from both regions.

The Spanner leader lives in Iowa. So writes from Iowa replicate synchronously to South Carolina before committing. Writes from South Carolina travel to Iowa, get committed, then come back.

Operation	Regional Iowa	nam6 Iowa	nam6 S.Carolina	nam6 S.Carolina + DR
Create ×1	90	53	11	11
Create ×4 parallel	270	203	45	45
Get ×1	108	116	14	16
Get ×4 parallel	481	577	60	64
Mixed ×4	403	327	49	53
Watch latency	~30ms	42ms	131ms	196ms

DR = --spanner-read-location=us-east1 — directed reads to the local South Carolina replica.

What this tells you:

Writes from Iowa get ~40% slower — that's the cost of synchronous replication to South Carolina. From South Carolina writes are 8× slower — each write travels Iowa→S.Carolina twice.

Directed reads improve read latency by 7-14% from South Carolina — reads go to the local replica instead of Iowa. The improvement is modest because writes still dominate the mixed workload, and Watch latency actually gets worse (Change Stream cursors still follow the leader path).

The practical conclusion: put your spanner-etcd replicas in the same region as the Spanner leader. If you need RPO=0 and must run replicas far from the leader, use --spanner-read-location to at least get reads locally. But writes will always pay the cross-region round-trip.

Kubernetes validation

We ran Kubernetes v1.33.12 (kubeadm, external etcd = spanner-etcd) for 24 hours straight:

Rolling deployments scaled 1–10 replicas every 2 minutes
ConfigMap churn every 3 minutes
cert-manager running concurrently
57 active Watch streams throughout

Results: zero crashes, zero data loss, zero unimplemented errors. The Kubernetes node stayed Ready the entire time.

We also tested with 22 production Java/Kotlin microservices (Vert.x + jetcd) on GKE. Auth token expiry, pod kill, Watch stream migration — all clean.

What we didn't build (and why)

Auth RBAC (UserAdd, RoleAdd, GrantPermission) — Kubernetes doesn't use it. The API server manages its own RBAC. We implement Authenticate (username/password → token) because kubeadm requires it, but the full RBAC surface isn't needed.

Defrag / Snapshot — Spanner manages storage automatically. These operations don't have a meaningful equivalent.

Sub-10ms Watch latency — if you need this, spanner-etcd is the wrong tool. Change Streams have inherent latency. For most Kubernetes operations this doesn't matter — the API server isn't latency-sensitive to etcd Watch at the millisecond level.

What surprised us

The covering index made a bigger difference than the PCT revision change. We expected the write bottleneck removal to be the headline. It was. But the read path optimization nearly doubled read throughput and nearly tripled mixed workload numbers. Sometimes the boring infrastructure work matters more than the clever architectural idea.

100 PU is genuinely enough for most clusters. We expected a linear relationship between PU and performance. Instead we found that network latency dominates and Spanner CPU is barely touched. The PU floor matters for parallel writes, but a small cluster doesn't need 1000 PU.

Non-leader regions are expensive. Iowa→South Carolina adds ~80ms round-trip. In a multi-region setup, where you place your replicas relative to the Spanner leader matters a lot more than how many PUs you provision.

Try it

docker run --rm \
  -e SPANNER_DATABASE=projects/P/instances/I/databases/D \
  -p 2379:2379 \
  ghcr.io/n0rm4l-me/spanner-etcd:v0.1.0

Or with kubeadm:

etcd:
  external:
    endpoints:
      - http://spanner-etcd:2379

GitHub: github.com/n0rm4l-me/spanner-etcd