Originally published on kuryzhev.cloud
A team applies a default-deny policy to a staging namespace on Friday afternoon, and within seconds every pod starts logging "temporary failure in name resolution". Or the opposite happens: the policy applies cleanly, nothing breaks, and nothing is actually blocked either. A NetworkPolicy baseline sounds like a simple YAML exercise, but the documented behavior has enough sharp edges that the same few mistakes keep appearing. This retrospective walks through those mistakes as patterns, not as a report of any specific outage.
Context: what a NetworkPolicy baseline is supposed to do
By default, Kubernetes pods accept traffic from any other pod in the cluster and can reach any destination. Namespaces are an organizational boundary, not a network one. A NetworkPolicy baseline changes that default: each namespace starts from "deny everything", and teams add narrow allow rules on top.
Three properties of the networking.k8s.io/v1 API shape everything that follows:
- Policies are allow-only and additive. There is no deny rule; the union of all policies selecting a pod defines what is permitted.
- A pod is only isolated for a direction (ingress or egress) once at least one policy selects it for that direction.
- The API object is only a declaration. Enforcement is the job of the CNI plugin or network provider.
A minimal baseline usually has two parts: a default-deny policy for both directions, and an explicit allowance for cluster DNS. The example below shows the shape. Verify the DNS labels against your own cluster, since they depend on the distribution.
# Baseline for one namespace: deny all ingress and egress for every pod
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: team-a
spec:
podSelector: {} # empty selector = every pod in the namespace
policyTypes:
- Ingress
- Egress
---
# Allow DNS lookups to CoreDNS in kube-system
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: team-a
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system # set automatically by the control plane (v1.21+)
podSelector:
matchLabels:
k8s-app: kube-dns # CoreDNS label in most distributions
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53 # needed for large responses and truncation fallback
Common failure 1: default-deny without a DNS and platform exception
The most typical failure is applying the deny policy first and discovering afterward that egress includes DNS. With policyTypes: [Egress] and no egress rules, pods cannot resolve service names. The symptom looks like an application bug: connection errors, readiness probes that depend on external calls failing, startup crashes on config fetches.
The fix is the DNS allowance shown above, but there are variations that catch people out. Watch out for NodeLocal DNSCache: when it is deployed, pods may send queries to a node-local link-local address rather than the CoreDNS service IP. The node-local-dns pods typically run with hostNetwork, so a namespace-and-pod selector may not match them, and an ipBlock rule for the link-local address may be needed. Whether that works also depends on the CNI. Check your DNS configuration (/etc/resolv.conf inside a pod) and consult your CNI documentation before assuming the CoreDNS rule is sufficient.
Other platform dependencies tend to surface the same way. Operators and controllers need egress to the Kubernetes API server. Metrics scrapers need ingress from a monitoring namespace. Service meshes need to reach their control planes. API server traffic is a common trap because the control plane usually sits outside the pod network. Pod and namespace selectors do not match it, so it typically needs an ipBlock rule. Whether that rule should target the kubernetes Service IP or the actual endpoint IPs depends on how your CNI orders address translation and policy evaluation. A deny baseline needs a short, reviewed list of these platform exceptions, ideally delivered by the same mechanism that creates the namespace, so they never lag behind.
Common failure 2: policies that exist but are never enforced
The Kubernetes API server accepts a NetworkPolicy object regardless of whether anything enforces it. The documentation is explicit that creating the resource without a supporting network plugin has no effect. A cluster using a CNI without policy support, or a managed cluster where the policy feature has not been enabled, will happily store the baseline while traffic flows freely.
This is the dangerous failure because it produces a green dashboard and a false sense of isolation. Reviewing the YAML alone cannot detect it; a traffic test is what confirms enforcement. Watch out for managed platforms where enforcement is an opt-in add-on or a separate component, and where the behavior may differ by version. Check your provider's documentation rather than assuming.
A simple connectivity probe turns the baseline into something testable. For example, run a throwaway pod in the isolated namespace and try to reach a service in another namespace. Because the baseline denies egress from team-a, the expected result is a timeout, not a response. Verify with a command like the one below, adjusting the target to your environment.
# Expect a timeout (exit code non-zero) if the egress baseline is enforced
kubectl -n team-a run np-test --rm -it --restart=Never \
--image=busybox:stable -- \
wget -qO- -T 3 http://web.team-b.svc.cluster.local # -T 3 = 3 second timeout
Note that a DNS failure also exits non-zero. Busybox reports it as a "bad address" error rather than a timeout, so read the output, not just the exit code. Test against a pod IP as well, so you know which layer is blocking. This probe exercises team-a egress. To verify ingress isolation for a namespace, run the probe from a namespace that does not deny egress and target a pod in the protected namespace.
Common failure 3: selector semantics that quietly widen access
The third pattern is subtle: YAML indentation changes the meaning of a rule. Inside a from or to list, a namespaceSelector and podSelector in the same list item are combined with AND. Placed as separate items, they are combined with OR. Take a rule intended as "pods labeled app.kubernetes.io/name=prometheus in the monitoring namespace". Split into two items, it becomes "every pod in the monitoring namespace, plus every pod with that label in the policy's own namespace".
# Intended: only Prometheus pods from the monitoring namespace
ingress:
- from:
- namespaceSelector: # one list item, two selectors = AND
matchLabels:
kubernetes.io/metadata.name: monitoring
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
# Risky variant: a second dash before podSelector makes it OR
Related traps include relying on custom namespace labels that anyone with namespace edit rights can change, and using ipBlock for in-cluster traffic, where pod IPs may be rewritten or ephemeral. Watch out for hostNetwork pods and node-originated traffic too. The documentation notes that behavior for hostNetwork pods is implementation-dependent, and that a pod cannot block access from its own node. That is why kubelet health probes generally keep working, but verify rather than assume. For namespace matching, prefer the kubernetes.io/metadata.name label. The control plane sets it to the namespace name, and it cannot be changed to a different value.
Safer operating pattern
The pattern that avoids all three failures is boring and incremental. Prove enforcement first, ship the deny and DNS policies together, then add allow rules per application with review. Treat the policies as code, applied by the same pipeline that provisions namespaces, so no namespace exists in an unprotected state.
A practical checklist, in order:
1. Confirm the CNI enforces NetworkPolicy (read provider docs, then run a deny test)
2. Inventory platform dependencies: DNS, API server, metrics, mesh, ingress controller
3. Apply default-deny + DNS allow together, never deny alone
4. Add per-app allow policies using kubernetes.io/metadata.name and stable pod labels
5. Review every from/to block for AND vs OR (selectors in one item vs separate items)
6. Run automated allow and deny connectivity tests in CI or post-deploy
7. Use your CNI's flow logs or policy-verdict logs to find denied traffic before tightening
8. Re-test after CNI, Kubernetes, or node image upgrades
For clusters with many namespaces, also look at the cluster-scoped AdminNetworkPolicy and BaselineAdminNetworkPolicy APIs from SIG Network's Network Policy API subproject. They aim to let platform teams set a baseline that namespace owners cannot override. However, they are distributed as CRDs rather than built into Kubernetes, are still pre-GA, and work only with CNIs that implement them. Check the project's current status and your CNI's support before depending on them.
Finally, remember what a baseline does not cover: it is layer 3/4 filtering, not authentication or encryption. It pairs well with RBAC, Pod Security Admission, and mutual TLS, not instead of them. The authoritative reference for semantics and current behavior is the Kubernetes NetworkPolicy documentation. More practical Kubernetes and security notes are available at kuryzhev.cloud.
Top comments (0)