DEV Community

Atul Anand Oraon
Atul Anand Oraon

Posted on

Troubleshooting Crossplane Installation Failure

#kubernetes #crossplane #debugging

Situation

While setting up Crossplane on a local kind Kubernetes cluster for the GCP GitOps project, the initial Helm installation failed:

Error: INSTALLATION FAILED: context deadline exceeded
Enter fullscreen mode Exit fullscreen mode

The Crossplane Helm repository was working correctly and the Crossplane image was successfully pulled, so the failure appeared to be occurring after Kubernetes created the Crossplane pods.

Task

Determine why Crossplane could not become ready and restore the cluster to a healthy state without unnecessarily recreating the Kubernetes cluster or blindly reinstalling Crossplane.

Action

1. Inspected Crossplane pods 

kubectl get pods -n crossplane-system
Enter fullscreen mode Exit fullscreen mode

Both Crossplane components were stuck in their init containers:

crossplane                0/1   Init:Error
crossplane-rbac-manager   0/1   Init:Error
Enter fullscreen mode Exit fullscreen mode

2. Inspected Kubernetes events 

kubectl get events -n crossplane-system --sort-by=.lastTimestamp
Enter fullscreen mode Exit fullscreen mode

The Crossplane image had been pulled successfully, but both init containers repeatedly crashed with:

Back-off restarting failed container crossplane-init
Enter fullscreen mode Exit fullscreen mode

This ruled out an image-pull problem.

3. Inspected init-container logs 

kubectl logs -n crossplane-system \
  crossplane-... \
  -c crossplane-init
Enter fullscreen mode Exit fullscreen mode

The critical error was:

failed to get server groups:
Get "https://10.96.0.1:443/api":
dial tcp 10.96.0.1:443: i/o timeout
Enter fullscreen mode Exit fullscreen mode

The RBAC manager showed the same failure.

This indicated that Crossplane itself was unable to communicate with the Kubernetes API through the cluster's Kubernetes Service.

4. Investigated Kubernetes Service networking

Checked the Kubernetes Service:

kubectl get svc kubernetes
Enter fullscreen mode Exit fullscreen mode

Result:

kubernetes   ClusterIP   10.96.0.1   443/TCP
Enter fullscreen mode Exit fullscreen mode

Checked its endpoint:

kubectl get endpoints kubernetes
Enter fullscreen mode Exit fullscreen mode

Result:

kubernetes   172.18.0.4:6443
Enter fullscreen mode Exit fullscreen mode

The API server was running and had a valid endpoint.

5. Investigated kube-proxy 

kubectl get pods -n kube-system
Enter fullscreen mode Exit fullscreen mode

Found:

kube-proxy-hdxsw   0/1   Error
Enter fullscreen mode Exit fullscreen mode

Further inspection showed:

kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=100
Enter fullscreen mode Exit fullscreen mode

Critical error:

command failed
err="failed complete: too many open files"
Enter fullscreen mode Exit fullscreen mode

This explained why Kubernetes Service networking was failing.

6. Independently verified the networking failure

A pod was used to access the Kubernetes API:

kubectl run api-test \
  --image=curlimages/curl \
  --restart=Never \
  --rm -it \
  -- curl -k --connect-timeout 5 \
  https://kubernetes.default.svc/api
Enter fullscreen mode Exit fullscreen mode

Before fixing kube-proxy:

curl: (28) Resolving timed out after 5005 milliseconds
Enter fullscreen mode Exit fullscreen mode

This confirmed that the problem was not specific to Crossplane.

7. Investigated host resource limits

Checked the host's inotify limits:

cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instances
Enter fullscreen mode Exit fullscreen mode

Initial values:

65536
128
Enter fullscreen mode Exit fullscreen mode

Kind documents "too many open files" as potentially being caused by exhausted inotify resources and recommends increasing these limits.

The system-wide file descriptor table was not exhausted:

cat /proc/sys/fs/file-nr
Enter fullscreen mode Exit fullscreen mode

showed:

12726   0   9223372036854775807
Enter fullscreen mode Exit fullscreen mode

Therefore, the issue was investigated as an inotify/file-watch resource-limit problem rather than general system-wide file descriptor exhaustion.

8. Increased inotify limits 

sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl fs.inotify.max_user_instances=512
Enter fullscreen mode Exit fullscreen mode

Verified:

fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512
Enter fullscreen mode Exit fullscreen mode

9. Restarted kube-proxy 

kubectl delete pod -n kube-system -l k8s-app=kube-proxy
Enter fullscreen mode Exit fullscreen mode

The replacement pod started successfully:

kube-proxy-z4dkd   1/1   Running
Enter fullscreen mode Exit fullscreen mode

Its logs no longer contained the previous too many open files failure.

10. Re-tested Kubernetes API connectivity

Ran the same API test again:

kubectl run api-test \
  --image=curlimages/curl \
  --restart=Never \
  --rm -it \
  -- curl -k --connect-timeout 5 \
  https://kubernetes.default.svc/api
Enter fullscreen mode Exit fullscreen mode

This time the request reached the Kubernetes API and returned:

403 Forbidden
User "system:anonymous" cannot get path "/api"
Enter fullscreen mode Exit fullscreen mode

The 403 was expected because the test pod was making an unauthenticated request. The important change was:

Before: i/o timeout
After:  HTTP 403
Enter fullscreen mode Exit fullscreen mode

This proved that Kubernetes Service networking had been restored.

11. Restarted Crossplane

The original Crossplane pods had been created while kube-proxy was broken, so they were recreated:

kubectl delete pod -n crossplane-system \
  crossplane-595f7997d8-rjfxn \
  crossplane-rbac-manager-584bbb6c67-pcdrr
Enter fullscreen mode Exit fullscreen mode

The replacement pods became healthy:

crossplane                1/1   Running
crossplane-rbac-manager   1/1   Running
Enter fullscreen mode Exit fullscreen mode

12. Verified Crossplane CRDs 

kubectl get crds | grep crossplane
Enter fullscreen mode Exit fullscreen mode

Multiple Crossplane CRDs were successfully installed, including:

compositeresourcedefinitions.apiextensions.crossplane.io
compositions.apiextensions.crossplane.io
managedresourcedefinitions.apiextensions.crossplane.io
providerrevisions.pkg.crossplane.io
providers.pkg.crossplane.io
Enter fullscreen mode Exit fullscreen mode

13. Reconciled Helm's release state

The original Helm installation had timed out while waiting for Crossplane, so Helm recorded the release as:

STATUS: failed
Enter fullscreen mode Exit fullscreen mode

Even though Kubernetes resources had subsequently recovered.

The existing release was upgraded rather than reinstalled:

helm upgrade crossplane \
  crossplane-stable/crossplane \
  --namespace crossplane-system \
  --version 2.3.4 \
  --wait \
  --timeout 5m
Enter fullscreen mode Exit fullscreen mode

Helm then reported:

STATUS: deployed
REVISION: 2
Enter fullscreen mode Exit fullscreen mode

Final Crossplane state:

crossplane                1/1   Running
crossplane-rbac-manager   1/1   Running
Enter fullscreen mode Exit fullscreen mode

Result

Crossplane was successfully installed and healthy.

Final verification:

Helm release:       deployed
Crossplane:         1/1 Running
RBAC manager:       1/1 Running
Crossplane CRDs:    installed
Kubernetes API:     reachable from pods
kube-proxy:         1/1 Running
Enter fullscreen mode Exit fullscreen mode

Root Cause

The immediate failure was kube-proxy crashing with too many open files, which broke Kubernetes Service networking.

That prevented Crossplane's init containers from reaching the Kubernetes API through:

https://10.96.0.1:443
Enter fullscreen mode Exit fullscreen mode

The issue was resolved by increasing the host's inotify resource limits and restarting kube-proxy.

Key Debugging Lesson

The initial error was:

Helm: context deadline exceeded
Enter fullscreen mode Exit fullscreen mode

but that was only the symptom.

The actual debugging chain was:

Helm timeout
    ↓
Crossplane pods not ready
    ↓
Crossplane init container failing
    ↓
Kubernetes API request timing out
    ↓
Kubernetes Service networking broken
    ↓
kube-proxy CrashLoopBackOff
    ↓
"too many open files"
    ↓
inotify resource limits
    ↓
increase limits + restart kube-proxy
    ↓
Service networking restored
    ↓
Crossplane becomes healthy
Enter fullscreen mode Exit fullscreen mode

This reinforced the importance of troubleshooting Kubernetes failures from the bottom up:

Pod status → Events → Container logs → Cluster networking → Node/system resources → Fix → Independent verification.

Top comments (0)