DEV Community

Cover image for 5 Istio misconfigurations that istioctl analyze won't catch
Petr Petrenko
Petr Petrenko

Posted on

5 Istio misconfigurations that istioctl analyze won't catch

#kubernetes #servicemesh #istio #devops

istioctl analyze is the go-to tool for validating Istio configuration. It catches unknown hosts, missing Services, malformed specs. Good tool.

But there is a class of problem it does not catch: configurations that parse correctly but break traffic at runtime. No schema violation, no warning, just silent failures or hard-to-trace 503s.

Here are five I keep running into.

1. No mTLS enforcement

A namespace with no PeerAuthentication runs in PERMISSIVE mode by default: Istio accepts both plaintext and mTLS traffic. Your sidecar can encrypt, but nothing forces it to.

istioctl analyze is fine with this. No PeerAuthentication at all is a valid configuration.

The problem shows up when you assume mTLS is on and it isn't. For example, when a new namespace gets created and nobody adds the policy.

What you need:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT
Enter fullscreen mode Exit fullscreen mode

Without this, plaintext traffic enters your namespace silently.

2. VirtualService with retries but no timeout 

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: payment-service
spec:
  http:
  - retries:
      attempts: 3
      perTryTimeout: 5s
    route:
    - destination:
        host: payment-service
Enter fullscreen mode Exit fullscreen mode

This looks reasonable: three retry attempts, five seconds each. But there is no top-level timeout. If the upstream is slow or stuck, each attempt waits the full 5s and all three fire. One slow request from the client turns into up to 15 seconds of upstream load.

In a chain of services, this compounds. Three hops with the same config and a stuck dependency: 45 seconds of retry storm, all traced back to one request.

What you need:

    timeout: 10s
    retries:
      attempts: 3
      perTryTimeout: 3s
Enter fullscreen mode Exit fullscreen mode

istioctl analyze has no opinion on retry/timeout combinations.

3. DestinationRule with no outlier detection 

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: order-service
spec:
  host: order-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
Enter fullscreen mode Exit fullscreen mode

This DestinationRule is valid. But without outlier detection, there is no automatic ejection of pods that are degraded but still running.

Kubernetes readiness probes handle the obvious case: a pod that crashes or fails its health check gets removed from Endpoints. But a pod that is alive and passing its health check while returning 500s or responding very slowly stays in rotation. Outlier detection catches exactly this: it watches observed error rates, not just liveness.

Add this:

    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
Enter fullscreen mode Exit fullscreen mode

istioctl analyze has no opinion on missing outlier detection.

4. Running pods without an Istio sidecar

A pod without istio-proxy is outside the mesh. No mTLS, no telemetry. This can happen when:

  • A namespace has injection enabled but a specific Deployment has sidecar.istio.io/inject: "false"
  • A pod was created before injection was enabled and was never restarted
  • An operator or Job runs with injection disabled and nobody noticed

istioctl analyze won't flag this. The pod is running, the config is valid.

This is especially painful because the pod will still receive traffic, but it won't be mTLS-encrypted, and VirtualService routing rules won't apply to outbound traffic it initiates.

5. EnvoyFilter without a workloadSelector 

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: add-header
  namespace: production
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    ...
Enter fullscreen mode Exit fullscreen mode

No workloadSelector. This EnvoyFilter applies to every pod in the namespace. If it is in istio-system, it applies to every pod in the entire mesh.

EnvoyFilters are the most powerful and most dangerous Istio primitive. A misconfigured patch pushed without a selector has taken down whole clusters.

istioctl analyze validates the patch structure, not the blast radius.

Catching these automatically

I built meshscan to catch exactly this class of problem. It reads your cluster's Istio resources and reports semantic misconfigurations: things that are syntactically valid but operationally dangerous.

$ meshscan -n production

meshscan report: namespace production
4 issues: 1 critical, 2 high, 1 medium

[CRITICAL] PeerAuthentication
  check: mtls-enforcement
  no PeerAuthentication found; namespace defaults to PERMISSIVE

[HIGH] DestinationRule/order-service
  check: outlier-detection
  no outlier detection configured

[HIGH] VirtualService/payment-service
  check: retry-without-timeout
  retries.attempts=3 but no timeout set (unbounded retry amplification possible)

[MEDIUM] VirtualService/order-service
  check: missing-dr
  routes to order-service but no DestinationRule exists
Enter fullscreen mode Exit fullscreen mode

It scans a single namespace or the whole cluster with -A. JSON output for CI pipelines. No cluster-side install needed: just a kubeconfig and list RBAC permissions.

go install github.com/n0rm4l-me/meshscan/cmd/meshscan@latest
Enter fullscreen mode Exit fullscreen mode

The two tools complement each other. Run istioctl analyze for schema and config validation. Run meshscan for runtime semantic issues.

Have you hit any of these in production? Curious what other silent failure modes people have run into.

Top comments (0)