Kubernetes Zero Trust: Lesson from a Missing Namespace Tag

This article, originally published on devopsstart.com, dives into a critical, yet often overlooked, aspect of Kubernetes zero-trust networking. If you've ever dealt with silent network failures after deploying new namespaces, this one's for you.

Problem

You deploy a new namespace in your Kubernetes cluster, apply your standard zero trust policies, and then discover that pods in the new namespace cannot reach the monitoring stack. HTTP connections time out. DNS resolves fine. Endpoint slices show healthy backends. But traffic dies silently. You check the CNI policy status and see zero active policies for that namespace. The root cause is almost certainly a missing namespace label. In Kubernetes zero trust networking, labels on namespaces are not cosmetic metadata. They are first-class identity attributes that policy engines like Cilium, Calico, and Istio rely on to group pods into security zones. When a namespace lacks the required label, the policy engine cannot match it, and every pod inside inherits a default-deny posture that blocks all traffic.

Root Causes

Three specific causes produce this failure mode, and they often overlap.

1. Missing or Incorrect Namespace Label

The most common cause. Your CiliumNetworkPolicy or Calico NetworkPolicy uses matchLabels to select namespaces by a key like ns or environment. If the new namespace team-b has no ns label, the policy engine sees zero matching namespaces. The policy evaluates against the empty set and applies to nothing. Cilium and Calico both rely on label-based identity: a pod inherits the labels of its namespace plus its own pod labels. If the namespace lacks the expected key, the pod gets an identity that no policy rule targets.

2. Label Drift After Initial Creation

A namespace may have the correct label at creation time, but a subsequent kubectl label command or automated process can remove it. Kubernetes allows removing labels without any warning. If a critical label disappears, existing policies immediately stop matching that namespace. Traffic that was flowing for weeks suddenly breaks. This is insidious because no resource event triggers an alert as the label simply vanishes.

3. Custom Annotation Mismatch

Some CNIs and service mesh solutions use annotations instead of labels for policy targeting. For example, Istio can use sidecar.istio.io/inject annotations to decide which pods to inject Envoy sidecars. If you use custom annotations (for example, ns-group: monitoring) in your NetworkPolicy selector, a missing annotation causes the same silent failure. The policy engine checks annotations on the namespace object, finds none, and skips it.

Solution

The fix takes five seconds. The diagnosis takes five minutes if you know the commands.

Step 1: Verify the Namespace Labels

Run this command to inspect the namespace object:

$ kubectl get namespace team-b -o yaml

Look at the metadata.labels section. If you expect ns: team-b and see no ns key, you have found the root cause. If you see ns: team-b but the policy uses a different key like environment: production, the label exists but does not match.

Step 2: Check CNI-Specific Policy Status

For Cilium, list all endpoints and their identities:

$ cilium endpoint list

Look at the IDENTITY column for pods in team-b. Cilium folds namespace labels into the endpoint identity: the built-in namespace shows as k8s:io.kubernetes.pod.namespace=team-b, and a custom ns label shows as k8s:io.cilium.k8s.namespace.labels.ns=team-b. If that custom-label entry is absent, the namespace label is missing. For Calico, inspect the workload endpoints:

$ calicoctl get workloadendpoint -n team-b

If no workload endpoint exists or the profile name is generic (for example, knp.default), the policy is not applying.

Step 3: Test Connectivity with Explicit Verification

Run a connectivity test from a pod in team-b to a service in the monitoring namespace:

$ kubectl exec -it deploy/team-b-app -- curl -m 3 http://prometheus.monitoring.svc.cluster.local:9090

Expected result: connection timeout or no route to host. Then run the same test after a namespace label change.

Step 4: Add the Missing Label

Apply the correct label that your policy expects:

$ kubectl label namespace team-b ns=team-b --overwrite

Give the policy engine a few seconds to reconcile: Cilium recomputes endpoint identities shortly after the label change, and Calico reconciles its datapath soon after. Rerun the connectivity test. Traffic should now flow.

Step 5: Confirm Policy Activation

For Cilium, verify policy enforcement:

$ cilium endpoint list | grep team-b

You should see an identity that includes k8s:io.cilium.k8s.namespace.labels.ns=team-b (the namespace label) alongside the built-in k8s:io.kubernetes.pod.namespace=team-b. For Calico, list the active network policies:

$ calicoctl get networkpolicy -n team-b

You should see your policies listed.

Step 6: Verify with a Generic NetworkPolicy Check

List all NetworkPolicies across all namespaces:

$ kubectl get networkpolicy -A

Confirm that a policy exists that targets team-b. If none exists, your policies may rely solely on pod labels and not namespace labels. In that case, the fix is either adding a namespace selector to your existing policy or applying pod labels that match.

Prevention

Prevent this failure from recurring with three measures.

1. Enforce Labels at Namespace Creation

Use Kyverno or OPA/Gatekeeper to reject namespace creation without required labels. A Kyverno ClusterPolicy can require a ns label that matches the namespace name:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-ns-label
spec:
  rules:
  - name: require-ns
    match:
      any:
      - resources:
          kinds:
          - Namespace
    validate:
      message: "Namespace must have label ns equal to the namespace name."
      pattern:
        metadata:
          labels:
            ns: "{{ request.object.metadata.name }}"

2. Prevent Label Drift

Use admission controllers to block mutation of critical labels. Kyverno can also validate label updates:

- name: block-ns-label-removal
  match:
    any:
    - resources:
        kinds:
        - Namespace
  preconditions:
    all:
    - key: "{{ request.operation }}"
      operator: In
      value:
      - UPDATE
  validate:
    message: "Removing the label 'ns' is forbidden."
    deny:
      conditions:
        any:
        - key: "{{ request.object.metadata.labels.ns }}"
          operator: IsNull

3. Audit Namespace Labels in CI/CD

Before deploying to production, run a check on every namespace:

$ kubectl get namespace -o json | jq -r '.items[] | select(.metadata.labels.ns == null) | .metadata.name'

If the output is non-empty, fail the pipeline and alert the team. Integrate this into your pre-deployment validation stage alongside existing checks like those covered in our guide to Kubernetes troubleshooting.

A missing namespace label is the number one hidden cause of "zero trust not working" in Kubernetes. Add it, verify it, and automate its enforcement.