DEV Community

Varsharani Hawanna
Varsharani Hawanna

Posted on

CloudNativePG Admission Webhook Timeout on GKE Autopilot: Root Cause, Troubleshooting and Production Fix

#cloud #database #infrastructure #kubernetes

Solving CloudNativePG Admission Webhook Timeouts on GKE Autopilot

Tags: kubernetes gke cloudnativepg devops postgresql

Originally published on the IBM Community and republished here to make the content accessible to a wider technical audience.

Introduction

Admission webhooks are a critical component of Kubernetes operators. When a custom resource is created or modified, the Kubernetes API server communicates with the appropriate admission webhook to validate or mutate the request.

While deploying a PostgreSQL cluster using the CloudNativePG (CNPG) operator on Google Kubernetes Engine (GKE) Autopilot, we encountered an admission webhook timeout that prevented the PostgreSQL cluster from being created.

The issue was initially observed with EnterpriseDB (EDB) Postgres for Kubernetes, but our investigation showed that the problem was not specific to EDB or PostgreSQL.

Instead, the issue was related to how Kubernetes control-plane traffic interacts with restrictive NetworkPolicies in GKE Autopilot.

This article covers:

  • The symptoms we observed

  • How we investigated the issue

  • What we learned while working with Google Cloud Support

  • The root cause

  • The NetworkPolicy change that resolved the issue

  • Recommendations for troubleshooting similar webhook problems

Although CloudNativePG is used as the example, the same troubleshooting approach can apply to other Kubernetes operators and applications that rely on admission webhooks.

Environment

The issue was reproduced in the following environment:

  • Google Kubernetes Engine (GKE) Autopilot
  • Kubernetes v1.35
  • CloudNativePG (CNPG) Operator
  • Admission webhook enabled
  • PostgreSQL deployment using CloudNativePG/EnterpriseDB

The Problem: Admission Webhook Timeout

During PostgreSQL deployment, the Kubernetes API server failed while invoking the CloudNativePG admission webhook.

A typical error looked like this:

failed calling webhook "mcluster.cnpg.io":
Post "https://cnpg-webhook-service.<namespace>.svc:443/...":
dial tcp <webhook-ip>:9443: i/o timeout

As a result, the PostgreSQL custom resource could not be successfully created.

At first glance, the error appeared to indicate a problem with the webhook itself.

However, the webhook Pod was healthy.

That led us to investigate the connectivity path between the Kubernetes API server and the webhook.

Who Can Be Affected?

This issue is not limited to a specific PostgreSQL distribution.

It can potentially affect:

  • CloudNativePG Operator
  • EnterpriseDB Postgres for Kubernetes
  • Applications built on top of CloudNativePG
  • Other Kubernetes workloads using admission webhooks
  • Workloads protected by restrictive NetworkPolicies on GKE Autopilot

A useful symptom to look for is:

The webhook is healthy and reachable from application Pods, but the Kubernetes API server reports webhook timeout errors.

Initial Observations

We first verified the health of the components involved in the webhook communication.

The following components were working as expected:

  • CloudNativePG webhook Pod was running
  • Kubernetes Service was healthy
  • DNS resolution was working
  • Other Pods inside the cluster could connect successfully to the webhook

However, one important component could not communicate with the webhook:

The Kubernetes API server.

This changed the direction of our investigation.

Instead of focusing on the webhook implementation, we started investigating control-plane-to-webhook connectivity.

Why Was This Unexpected?

The same deployment worked successfully on a GKE Standard cluster.

Google Cloud documentation recommends allowing control-plane traffic using the cluster's masterIpv4CidrBlock when configuring firewall rules for admission webhooks.

However, with GKE Autopilot, the Kubernetes control plane is fully managed by Google, and the control-plane IP range is not exposed to the customer in the same way.

This raised an important question:

How should NetworkPolicies and firewall rules be configured when the Kubernetes API server's IP range is not available to the customer?

Investigation and Troubleshooting

We first verified the webhook, Service, DNS, and Pod-to-Pod connectivity. Everything was working as expected.

We then tried:

  • Changing the webhook port from 9443 to 443
  • Increasing the webhook timeout from 10 to 30 seconds

  • Reviewing and updating firewall rules
    None of these resolved the timeout.

  • Temporarily setting failurePolicy: Ignore
    This resolved the issue, but this is not a production solution.

This led us to investigate the Kubernetes NetworkPolicy, which ultimately revealed the root cause.

Working With Google Cloud Support

After exhausting the common troubleshooting approaches, we worked with the Google Cloud Support networking team to understand why the Kubernetes API server could not reach the admission webhook.

Together, we performed several tests:

  • Verified that the webhook Pod was healthy
  • Confirmed that the Service and DNS were functioning correctly
  • Performed packet-level connectivity testing
  • Tested multiple firewall configurations
  • Verified that the webhook was reachable from application Pods

Despite these checks, the Kubernetes API server continued to experience timeout errors.

During the investigation, the networking team suggested shifting our focus from VPC firewall rules to Kubernetes NetworkPolicy.

That turned out to be the key.

The NetworkPolicy Problem

Our existing NetworkPolicy contained restrictive from and to rules.

These rules explicitly define where traffic is allowed to originate from and where it is allowed to go.

On GKE Autopilot, the Kubernetes API server resides in a Google-managed control plane, outside the customer-visible network environment.

Because these control-plane addresses are not directly exposed to customer projects, it may not be possible to explicitly reference them in NetworkPolicy rules.

The networking team therefore recommended simplifying the NetworkPolicy by removing restrictive traffic selectors for the webhook communication and allowing the required traffic based primarily on the webhook port.

After removing the restrictive from and to clauses while continuing to allow the required webhook port, the Kubernetes API server was immediately able to communicate with the CloudNativePG admission webhook.

The PostgreSQL deployment then completed successfully.

Most importantly:

  • We did not modify the webhook implementation
  • We did not disable admission validation
  • We did not need to increase the webhook timeout
  • The PostgreSQL deployment completed successfully

This demonstrated that the issue was not caused by CloudNativePG itself.

Root Cause

The root cause was restrictive NetworkPolicy rules preventing the Kubernetes API server from reaching the admission webhook.

The important factor was the interaction between the NetworkPolicy and the Google-managed GKE Autopilot control plane.

Because the control plane exists outside the customer-visible network, NetworkPolicies that explicitly restrict traffic sources or destinations can unintentionally block admission webhook communication.

Production Solution

We resolved the issue by updating the Kubernetes NetworkPolicy.

Instead of restricting webhook traffic based on specific source or destination selectors, the policy was simplified to allow the required webhook communication while retaining the necessary namespace and DNS access.

After applying the updated policy:

  • The Kubernetes API server successfully reached the admission webhook
  • PostgreSQL deployment completed successfully
  • Admission validation remained enabled
  • No webhook configuration changes were required

Example NetworkPolicy

The following NetworkPolicy represents the configuration that resolved the webhook timeout in our GKE Autopilot environment.

It is provided as a reference implementation to demonstrate the important changes.

Important: Do not copy this policy directly into production without reviewing it. NetworkPolicy requirements vary by application and organization. Adapt the policy according to your security and networking requirements.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: unified-policy
  namespace: ${NAMESPACE}
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Allow traffic within the namespace
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ${NAMESPACE}
    # Allow Pod-to-Pod communication
    - from:
        - podSelector: {}
    # Allow admission webhook traffic on TCP 9443
    # Notice that there is no restrictive 'from'
    # selector for this rule.
    - ports:
        - protocol: TCP
          port: 9443
  egress:
    # Allow communication within the namespace
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ${NAMESPACE}
    # Allow Pod-to-Pod communication
    - to:
        - podSelector: {}
    # Allow DNS resolution
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: TCP
          port: 53
        - protocol: UDP
          port: 53
    # Allow webhook traffic on TCP 9443
    # Notice that there is no restrictive 'to'
    # selector for this rule.
    - ports:
        - protocol: TCP
          port: 9443
   # Allow outbound communication required by the application
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
Enter fullscreen mode Exit fullscreen mode

The key point is not the exact policy shown above.

The important takeaway is:

Make sure your NetworkPolicies do not unintentionally block communication between the Kubernetes API server and admission webhooks.

Conclusion

Admission webhook timeout errors on GKE Autopilot can be challenging to diagnose because the webhook itself may appear completely healthy and reachable from within the cluster.

In our case, the issue was not related to CloudNativePG or PostgreSQL. The root cause was the interaction between a restrictive Kubernetes NetworkPolicy and communication from the GKE Autopilot control plane to the admission webhook.

Once the restrictive traffic selectors were removed for the required webhook communication, the API server was able to reach the webhook and the PostgreSQL deployment completed successfully.

Although this article uses CloudNativePG as the example, the same troubleshooting approach can be applied to other Kubernetes operators and applications that rely on admission webhooks.

If you encounter a situation where:

Webhook Pod is healthy
↓
Service is healthy
↓
DNS works
↓
Pod-to-Pod connectivity works
↓
API server → webhook = timeout

Check your NetworkPolicies.

That simple check can save hours of troubleshooting.

Top comments (0)