DEV Community

Kader Khan
Kader Khan

Posted on

Calico Node Readiness Probe Failed Issues

#kubernetes #devops #networking #tutorial

🛠️ Resolving Calico Node Readiness Issues: A Practical Guide

🧩 Problem Overview

In Kubernetes clusters utilizing Calico as the networking solution, nodes may occasionally report a "not ready" status due to BIRD (Border Gateway Protocol) not initializing properly. This issue often stems from Calico's IP autodetection mechanism selecting an unintended network interface, leading to misconfigured BGP sessions which is impacting node to node communication.

🔍 Symptoms

  • Pods on the affected node cannot communicate with pods on other nodes.

  • The node's status is "not ready" in the Kubernetes cluster.

  • BIRD logs indicate errors like:

  bird: Unable to open configuration file /etc/calico/confd/config/bird.cfg: No such file or directory
  bird: Unable to open configuration file /etc/calico/confd/config/bird6.cfg: No such file or directory
Enter fullscreen mode Exit fullscreen mode

These errors suggest that BIRD cannot find its configuration files, often due to incorrect IP autodetection.

🧭 Root Cause

Calico's default IP autodetection method (first-found) may select an unintended interface, especially in nodes with multiple network interfaces. This misconfiguration can lead to BIRD being unable to establish proper BGP sessions, resulting in the node being marked as "not ready".

✅ Solution Approach

1. Identify the Correct Network Interface

Determine the appropriate network interface for Calico's BGP peering. Typically, this would be the primary network interface used for inter-node communication.

2. Set IP Autodetection Method Temporarily

To test the new configuration, set the IP_AUTODETECTION_METHOD environment variable on the Calico node DaemonSet:

kubectl set env daemonset/calico-node -n calico-system IP_AUTODETECTION_METHOD=interface=eth0
Enter fullscreen mode Exit fullscreen mode

Replace eth0 with the correct interface name identified in the previous step.

3. Verify the Configuration

Check the status of the Calico node pods to ensure they are running correctly:

kubectl get pods -n calico-system
Enter fullscreen mode Exit fullscreen mode

Additionally, inspect the logs of the Calico node pods to confirm that BIRD has initialized without errors:

kubectl logs -n calico-system calico-node-<pod-id>
Enter fullscreen mode Exit fullscreen mode

4. Set IP Autodetection Method Permanently

To make the change permanent, update the Calico Installation resource:

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
  namespace: tigera-operator
spec:
  calicoNetwork:
    nodeAddressAutodetectionV4:
      interface: eth0
Enter fullscreen mode Exit fullscreen mode

Apply the updated configuration:

kubectl apply -f <your-installation-file>.yaml
Enter fullscreen mode Exit fullscreen mode

This ensures that the specified interface is used for IP autodetection across all nodes in the cluster.

5. Restart Calico Node Pods

After applying the changes, restart the Calico node pods to apply the new configuration:

kubectl rollout restart daemonset/calico-node -n calico-system
Enter fullscreen mode Exit fullscreen mode

This command restarts the Calico node DaemonSet, ensuring that all pods pick up the new configuration.

🧪 Verification

After completing the steps above, verify that the node has transitioned to a "ready" state:

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

Ensure that the node in question is listed as "Ready".

Also, confirm that BIRD is running without errors:

kubectl exec -n calico-system calico-node-<pod-id> -- birdcl show status
Enter fullscreen mode Exit fullscreen mode

The output should indicate that BIRD is initialized and ready.

💡 Best Practices

  • Consistent Configuration: Ensure that the IP autodetection method is consistently configured across all nodes to avoid network inconsistencies.

  • Regular Monitoring: Regularly monitor the status of Calico node pods and BIRD to detect and resolve issues promptly.

  • Documentation: Document the network interfaces and configurations used for IP autodetection to facilitate troubleshooting and future configurations.

By following this approach, you can resolve Calico node readiness issues related to IP autodetection and ensure stable networking within your Kubernetes cluster.

Top comments (0)