DEV Community

Cover image for Fix Cert-Manager Conflict with EKS
Mohamed Radwan for AWS Community Builders

Posted on β€’ Edited on

6 1 1 1 1

Fix Cert-Manager Conflict with EKS

I was facing issue with multiple managed worker nodes running on EKS clusters.

The issue was appearing randomly in different nodes, I cannot access the pods or get the logs by kubectl.

x509: cannot validate certificate for 10.0.83.153 because it doesn’t contain any IP SANs 
Enter fullscreen mode Exit fullscreen mode

Kube API in the CloudWatch showing the following errors:

E0327 08:54:17.406029 11 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"error dialing backend: x509: cannot validate certificate for 10.0.83.153 because it doesn't contain any IP SANs"}: error dialing backend: x509: cannot validate certificate for 10.0.83.153 because it doesn't contain any IP SANs 
Enter fullscreen mode Exit fullscreen mode

After investigating the issue with the AWS EKS support team, we found that cert-manager-webhook is causing the issue.
Kubelet certificate chain is being used from cert-manager-webhook-ca.

Run the following command on the non-working node:

openssl s_client -connect localhost:10250 
Enter fullscreen mode Exit fullscreen mode
CONNECTED(00000003)
---
Certificate chain
 0 s:
   i:/CN=cert-manager-webhook-ca
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=
issuer=/CN=cert-manager-webhook-ca
---
Enter fullscreen mode Exit fullscreen mode

Run the following command on the working healthy node:

openssl s_client -connect localhost:10250 
Enter fullscreen mode Exit fullscreen mode
CONNECTED(00000003)
---
Certificate chain
 0 s:/O=system:nodes/CN=system:node:ip-10-0-31-151.eu-west-1.compute.internal
   i:/CN=kubernetes
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=/O=system:nodes/CN=system:node:ip-10-0-31-151.eu-west-1.compute.internal
issuer=/CN=kubernetes
---
Enter fullscreen mode Exit fullscreen mode

The cert-manager-webhook deployment uses port 10250 which is also used for kubelet.

The solution is change the port of cert-manager-webhook to 10260.

By setting webhook.securePort to 10260

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.10.0 \
  --set webhook.securePort=10260
Enter fullscreen mode Exit fullscreen mode

Sources:

https://cert-manager.io/docs/concepts/webhook/
https://cert-manager.io/docs/installation/compatibility/#aws-eks

Image of AssemblyAI tool

Transforming Interviews into Publishable Stories with AssemblyAI

Insightview is a modern web application that streamlines the interview workflow for journalists. By leveraging AssemblyAI's LeMUR and Universal-2 technology, it transforms raw interview recordings into structured, actionable content, dramatically reducing the time from recording to publication.

Key Features:
πŸŽ₯ Audio/video file upload with real-time preview
πŸ—£οΈ Advanced transcription with speaker identification
⭐ Automatic highlight extraction of key moments
✍️ AI-powered article draft generation
πŸ“€ Export interview's subtitles in VTT format

Read full post

Top comments (0)

Best Practices for Running  Container WordPress on AWS (ECS, EFS, RDS, ELB) using CDK cover image

Best Practices for Running Container WordPress on AWS (ECS, EFS, RDS, ELB) using CDK

This post discusses the process of migrating a growing WordPress eShop business to AWS using AWS CDK for an easily scalable, high availability architecture. The detailed structure encompasses several pillars: Compute, Storage, Database, Cache, CDN, DNS, Security, and Backup.

Read full post

πŸ‘‹ Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay