DEV Community: Algo7

Rename a Kubernetes PVC Without Losing Your Data: PersistentVolume Rebinding

Algo7 — Thu, 28 May 2026 16:04:29 +0000

Introduction

As an SRE working with Kubernetes, you'll occasionally need to rename a PersistentVolumeClaim (PVC) without losing the PersistentVolume (PV) and the data behind it. It comes up when you rename a StatefulSet, or just rename a Helm release for cosmetic reasons, like changing myservice to my-service. Because the PVC name is derived from that name in both cases, a simple redeploy provisions a new PVC, and with it a new, empty PV. If your old PV's persistentVolumeReclaimPolicy is Retain, your data stays unreferenced on the old PV (if it's Delete, the volume is gone the moment you delete the PVC).

To keep the data, your first instinct might be a migration: taking a VolumeSnapshot, running rsync between two mounts, or copying the data out to your local machine and back into the PV. But, in this very specific case, there's an easier way: a PV rebind. Instead of moving any data, you keep the PV as is and just re-point it at the new PVC.

Worth getting the vocabulary straight: this is a rebind, not a migration. Not a single byte moves. For block storage the volume ID stays the same; for NFS the same access point remains.

Important Notes
Step 1: Set the persistentVolumeReclaimPolicy to Retain
Step 2: Scale Down the StatefulSet / Deployment
Step 3: Delete the PVC
Step 4: Modify the Existing PV
Step 5: Apply the New PVC
Note on block-storage based CSIs on Managed Kubernetes
Automated Tooling

Important Notes

For dynamic PV provisioning: don't create the new PVC (or a StatefulSet with volumeClaimTemplates) until after you've re-pointed the existing PV in Step 4, or the provisioner races in and hands you a new, empty PV. If you already created the news resources, delete them along with the associated PVs and PVCs.
For static PV provisioning: well, just don't create a new PV. The whole point is to reuse the existing one.
If you're on ArgoCD, turn off auto-sync or it will keep reverting the changes you're making.

Step 1: Set the persistentVolumeReclaimPolicy to Retain

A PV provisioned by a StorageClass (SC) inherits that SC's reclaimPolicy at creation time. So if the SC's reclaimPolicy is Delete, the PV's persistentVolumeReclaimPolicy will be Delete too, and the moment you delete the PVC the underlying volume and your data are gone.

An SC's reclaimPolicy is immutable: even if you delete and recreate the SC, the change only applies to PVs provisioned afterward. The PV's own persistentVolumeReclaimPolicy, however, is mutable, which is exactly what we are going to change.

First, check the reclaim policy of your existing PV:

kubectl get pv <your-pv-name> -o jsonpath='{.spec.persistentVolumeReclaimPolicy}'

If it prints Retain, you're safe to move on. If it prints Delete, change it:

kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

# OR edit the same field interactively:
kubectl edit pv <your-pv-name>

Step 2: Scale Down the StatefulSet / Deployment

You can't delete a PVC while a pod is still using it: the pvc-protection finalizer blocks it. Scale the workload down first.

For Deployments:

kubectl scale deployment <deployment-name> --replicas=0 -n <your-namespace>

For StatefulSets:

kubectl scale statefulset <statefulset-name> --replicas=0 -n <your-namespace>

Step 3: Delete the PVC

Delete your existing PVC:

kubectl delete pvc <your-pvc-name> -n <your-namespace>

With Retain set in Step 1, the PV state changes to Released, keeping your data instead of reclaiming it.

Step 4: Modify the Existing PV

Delete two fields under spec.claimRef on the existing PV:

resourceVersion
uid

kubectl patch pv <your-pv-name> --type='merge' -p '{"spec":{"claimRef":{"resourceVersion":null,"uid":null}}}'

The uid binds the PV to the old PVC's specific instance; clearing it and the resourceVersion flips the PV from Released to Available and thus free to bind a new claim. You then point claimRef at the new PVC by changing the name (and namespace, if it moved):

kubectl patch pv <your-pv-name> --type='merge' -p '{"spec":{"claimRef":{"name":"new-claim-name","namespace":"new-namespace-name"}}}'

You can also do this interactively with kubectl edit pv <your-pv-name>, but becareful only touch the fields under claimRef. If you delete the PV's top-level metadata.uid, which is what I accidentally did, the kubectl edit will throw error: no original object found.

Step 5: Apply the New PVC

Now create the new PVC using kubectl, Helm, or whatever you use. Give it the same storageClassName and accessModes as the PV, a size no larger than the PV's capacity, and leave volumeName unset. Because the PV is already reserved for this exact name, the binder grabs it instead of provisioning a fresh one.

Note on Personal Experience: if your new PVC requests a larger size, the rebinding won't work and a new PV will provisioned by your SC. Smaller size will work, but you still get whatever the old PV size is which is misleading; so keep the manifest exactly the same as before.

Note on block-storage based CSIs on Managed Kubernetes

Block storage on managed Kubernetes, such as EKS with the EBS CSI driver, is usually AZ-bound. If the PV lives in availability-zone-a, you can't attach it to a pod scheduled in availability-zone-b. There you'd have to take the full migration path, which we won't cover today.

Automated Tooling

This process can also be automated using tools such as rename-pvc from STACKIT. It follows the same PV rebind procedure.

However, you still need to remember to scale down the workload and pause any GitOps reconciliation (e.g. ArgoCD auto-sync) during the operation.

K3s Update TLS SANs

Algo7 — Sat, 09 Mar 2024 21:02:44 +0000

Introduction

When I was following the official guide of adding an external cluster load balancer for K3s using HAProxy to have some HAs when connecting to the cluster. I ran into a problem: kubectl refuses to connect to the cluster due to some certificate issues:

couldn't get current server API group list: Get "https://balancer_ip:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate is valid for [list of control plan ips and cluster + Kube API server service ip], ::1, not load_balancer_ip

This is due to my load balancer's IP not being in the cluster's API server certificate. During cluster initialization and when adding new control plane node to the cluster, K3s automatically update this certificate to include the new control plane node. However, this is not the case when adding an external cluster load balancer.

Solution 1: Reinitialize the Cluster With --tls-san
Solution 2: Editing K3s Configuration File
- Step 1: Edit the Config File
- Step 2: Delete the Existing Kube API Server Cert
- Step 3: Restart K3s
- Step 4: Inform the DynamicListener About the Change
  - Alternative Method
Worker / Control Plane Node Reconfiguration (Optional)

Solution 1: Reinitialize the Cluster With --tls-san

According to the K3s official documentation, you can pass the --tls-san flag when initializing the cluster:

sh k3s_install.sh --tls-san additional_ip_1 --tls-san additional_ip_1 --cluster-init

However, my cluster is already running and I don't want to reinstall it.

Solution 2: Editing K3s Configuration File

The K3s configuration file is located under /etc/rancher/k3s/config.yaml on all control plane nodes. If the file doesn't exist, you have to create it manually.

Step 1: Edit the Config File

In the config file, you can simply add the following entries:

tls-san:
  - ip_1
  - hostname_1
  // additional entries

This can be done on any control plane node.

Step 2: Delete the Existing Kube API Server Cert

On any control plane node run

sudo kubectl -n kube-system delete secrets/k3s-serving

to delete the existing Kubernetes API server certificate.

This can be done anywhere as long as you can access the cluster's API server.

Step 3: Restart K3s

Restart K3s on the same control plane node:

sudo systemctl restart k3s

Step 4: Inform the DynamicListener About the Change

DynamicListener is a component of K3s that handles automatic updates/renewal of the API server certificate, including when new control plan nodes join the cluster.

According to one of the K3s contributor, Brad Davidson in one of the GitHub issue:

Dynamiclistener adds SANs for any hostname or IP address requested via a HTTP Host header or TLS SNI handshake. It is designed to allow you to add or remove servers without having to manually regenerate the certificate. The downside (as you noted) is that this allows for SAN stuffing.

The documentation about it is not very clear and the official README is still in progress. This is as far as I can go about it.

For each new hostname / IP you added to the config file, run the following command on the control plan node:

curl -k --resolve your_new_hostname_or_ip:6443:127.0.0.1  https://your_new_hostname_or_ip:6443/ping

This method is also mentioned in another GitHub issue comment here.

To confirm that the API server certificate has been updated, run the following command:

kubectl get secret/k3s-serving -n kube-system -o yaml

This can be done anywhere as long as you have access to the cluster's API server.

In the metadata.annotations part of the output, you should see your newly added hostname/ip as one of the annotations:

listener.cattle.io/cn-ur_hostnameor__ip: ur_hostname_or_ip

Alternative Method

I found one Medium post here that uses another method to update the DynamicListener.

After you delete the existing API server certificate on a control plane node using

sudo kubectl -n kube-system delete secrets/k3s-serving

You can move or delete a file called dynamic-cert.json located under /var/lib/rancher/k3s/server/tls on the same control plane node

sudo mv /var/lib/rancher/k3s/server/tls/dynamic-cert.json /tmp/dynamic-cert.json

BEFORE restarting K3s on the same control plane node.

I have tested both methods and they all worked.

Worker / Control Plane Node Reconfiguration (Optional)

Now we have to update our worker / control plane nodes to use the new endpoint.

The fastest way is to update the K3s systemd service file.

You will have to do this on each node that you want to reconfigure.

For control plane nodes the file is located at

/etc/systemd/system/k3s.service

For worker nodes the file is located at

/etc/systemd/system/k3s-agent.service

Update the value of the --server flag in the file to use the new hostname / IP of the balancer.

After making the changes, run sudo systemctl daemon-reload to reload the changes.

Finally run sudo systemctl restart k3s to restart the K3s daemon for the changes to take effect.

Install K3s on Proxmox Using Ansible

Algo7 — Tue, 30 Jan 2024 12:50:23 +0000

Introduction

This article will guide you through how to install K3s with Flannel VXLAN backend on Proxmox using Ansible. It is assumed that you have basic knowledge about Kubernetes, Proxmox, SSH, and Ansible.

This is my first post on dev.to, any constructive feedback is welcomed.

Introduction
Table Of Contents
Prerequisites
Proxmox Firewall Configuration
- Creating Proxmox Security Group
- Applying the Security Groups to your VMs
Ansible Setup
- Update your SSH config
- Directory Structure
- Ansible Config File
- Ansible Inventory
- The Enrollment Token
- The K3s Ansible Role
- The Playbook
Run the Playbook
Cluster Access
Resources

Prerequisites

Promox installed and running
Ansible installed on your local machine
At least 4 VMs (3 control plane nodes + 1 worker node)
- OS: Debian-based OSes, preferably the latest Ubuntu LTS
- SSH installed and configured using Public Key Auth
kubectl installed locally
SSH installed locally

Proxmox Firewall Configuration

When you create VMs on Proxmox, the firewall is disabled by default. If you want to use the VM firewall, please make sure you opened the required ports.

Below are the inbound rules for K3s nodes from the official documentation:

For more information: https://docs.k3s.io/installation/requirements#inbound-rules-for-k3s-nodes

We don't need all of these ports opened because we are not using the Flannel Wireguard backend and Spegel registry.

Creating Proxmox Security Group

As we will be applying the same firewall rules to multiple VMs, it will be easier if we create a security group in Proxmox and apply the security group to all the target VMs.

We will be creating 2 Security Groups in Proxmox:

k3s: Ports that need to be opened on all nodes
k3s_server: Ports that only need to be opened on the control plane nodes

To create a security group in Proxmox:

Login to the Proxmox UI
Go to Datacenter on the left menu
Go to Firewall and then Security Group
Click on Create
Select the security group you just created and click on Add to start adding rules

For the k3s security group:

TCP 10250 for Kubelet metrics
UDP 8472 for Flannel VXLAN
TCP 22 for SSH

For the k3s_server security group:

TCP 6443 for K3s supervisor and Kubernetes API Server
TCP 2479 for HA with embedded etcd
TCP 2380 for HA with embedded etcd

Applying the Security Groups to your VMs

Click on your VM on from the left menu
Select Firewall
Click on the Insert: Security Group button
Select the security groups we just created and enable them

Ansible Setup

Update your SSH config

To streamline our process of working with Ansible, it is recommended that you add the target hosts to your SSH config so we can reference them using hostnames in the Ansible inventory file. I am using Ubuntu on my local machine, so the SSH config is located at ~/.ssh/config. The same should apply to all Linux/Unix OSes.

Example configuration

# Control plane 1
Host k3s-m1
  User ubuntu
  HostName 10.0.0.30
  IdentityFile ~/.ssh/your_pk

# Control plane 2
Host k3s-m2
  User ubuntu
  HostName 10.0.0.31
  IdentityFile ~/.ssh/your_pk

# Control plane 3
Host k3s-m3
  User ubuntu
  HostName 10.0.0.32
  IdentityFile ~/.ssh/your_pk

# Worker node 1
Host k3s-w1
  User ubuntu
  HostName 10.0.0.33
  IdentityFile ~/.ssh/your_pk

# Add more if needed

Directory Structure

.
├── ansible.cfg # project specific Ansible configuration
├── inventory
│   └── inventories.ini # Inventory information
├── k3s.yml # The playbook to install K3s
└── roles
    └── k3s
        ├── tasks
        │   └── main.yml # Ansible role to install K3s
        └── vars
            └── k3s-secrets.yml # K3s enrollment token

Ansible Config File

Path: project_root/ansible.cfg

We will create a custom Ansible configuration file named ansible.cfg at the root of the project directory, which will reference the inventory file.

Example configuration:

[defaults]
inventory = ./inventory/inventories.ini
# If you haven't connected to the K3s VMs before and they are not in
# your SSH `know_hosts` file, you will have to uncomment the
# following option for the playbook to not thrown an error:
# host_key_checking = False

Ansible Inventory

Path: project_root/inventory/inventories.ini

We will organize the target VMs using an Ansible inventory file.

# Initial master node setup for bootstrapping the K3S cluster.
# 'node_type' is used in the Ansible roles to identify and execute specific tasks for this node (see the role section).
[k3s_initial_master]
# k3s-m1 is the hostname we defined in our SSH config
k3s-m1 node_type=k3s_initial_master

# Additional master nodes for the K3S cluster.
[k3s_masters]
k3s-m2 node_type=k3s_master
k3s-m3 node_type=k3s_master
# Additional workers...

# Worker nodes for running containerized applications.
[k3s_workers]
k3s-w1 node_type=k3s_worker
k3s-w2 node_type=k3s_worker
k3s-w3 node_type=k3s_worker
# Additional workers...

# Group definition for simplified playbook targeting.
[k3s:children]
k3s_initial_master
k3s_masters
k3s_workers

The Enrollment Token

Path: project_root/roles/k3s/vars/k3s-secrets.yml

You can optionally encrypt the file using the ansible-vault encrypt path_to_file command

---
# You have to create a token yourself
k3s_token: "your_token"

The K3s Ansible Role

Path: project_root/roles/k3s/tasks/main.yml

# code: language=ansible
---
- name: Install K3S Requirements
  ansible.builtin.apt:
    update_cache: true
    pkg:
      - policycoreutils
    state: present

# To make sure the the role is idempotent. The tasks after this will only be executed if K3S hasn't been installed already.
- name: Check if K3S is already installed
  ansible.builtin.shell:
    cmd: 'test -f /usr/local/bin/k3s'
  register: k3s_installed
  failed_when: false

- name: Download K3s installation script
  ansible.builtin.uri:
    url: 'https://get.k3s.io'
    method: GET
    return_content: yes
    dest: '/tmp/k3s_install.sh'
  when: k3s_installed.rc != 0

- name: Import K3S Token
  ansible.builtin.include_vars:
    file: k3s-secrets.yml.vault
  when: k3s_installed.rc != 0

# Note that the node_type variable is set in the inventory file
- name: Execute K3s installation script [Initial Master Node]
  ansible.builtin.shell:
    cmd: 'sh /tmp/k3s_install.sh --token {{ k3s_token }} --disable=traefik --flannel-backend=vxlan --cluster-init'
  vars:
    k3s_token: '{{ k3s_token }}'
  args:
    executable: /bin/bash
  when: node_type | default('undefined') == 'k3s_initial_master' and k3s_installed.rc != 0

- name: Execute K3s installation script [Master Nodes]
  ansible.builtin.shell:
    cmd: 'sh /tmp/k3s_install.sh --token {{ k3s_token }} --disable=traefik --flannel-backend=vxlan --server https://{{ hostvars["k3s-m1"]["ansible_default_ipv4"]["address"] }}:6443'
  vars:
    k3s_token: '{{ k3s_token }}'
  args:
    executable: /bin/bash
  when: node_type | default('undefined') == 'k3s_master' and k3s_installed.rc != 0

- name: Execute K3s installation script [Worker Nodes]
  ansible.builtin.shell:
    cmd: 'sh /tmp/k3s_install.sh agent --token {{ k3s_token }} --server https://{{ hostvars["k3s-m1"]["ansible_default_ipv4"]["address"] }}:6443'
  vars:
    k3s_token: '{{ k3s_token }}'
  args:
    executable: /bin/bash
  when: node_type | default('undefined') == 'k3s_worker' and k3s_installed.rc != 0

The Playbook

Path: project_root/k3s.yml

# code: language=ansible
# K3S Ansible Playbook
---
- name: K3S
  hosts: k3s
  gather_facts: true
  roles:
    - role: k3s
      become: true

Run the Playbook

In your terminal, run ansible-playbook k3s.yml. If you have encrypted the k3s-secret.yml then you should run ansible-playbook k3s.yml --ask-vault-pass and enter the password.

Cluster Access

After the playbook has finished running. You can obtain the content of the cluster's kubeconfig by SSH into one of the master node and reading the content of /etc/rancher/k3s/k3s.yaml

More information: https://docs.k3s.io/cluster-access

To use the config locally, please remember to change the server: https://127.0.0.1:6443 property in the file to point to one of the master node.

Once you have copied the kubeconfig to your local machine, you can run kubectl get ns to list all the namespaces and test the connection. If you have multiple clusters locally, make sure you are selecting the the correct context.

Resources

The playbook and role configuration can be found here on my GitHub repo: https://github.com/algo7/homelab_ansible/tree/main/roles/k3s

DEV Community: Algo7

Rename a Kubernetes PVC Without Losing Your Data: PersistentVolume Rebinding

Introduction

Table of Contents

Important Notes

Step 1: Set the persistentVolumeReclaimPolicy to Retain

Step 2: Scale Down the StatefulSet / Deployment

Step 3: Delete the PVC

Step 4: Modify the Existing PV

Step 5: Apply the New PVC

Note on block-storage based CSIs on Managed Kubernetes

Automated Tooling

K3s Update TLS SANs

Introduction

Table of Contents

Solution 1: Reinitialize the Cluster With --tls-san

Solution 2: Editing K3s Configuration File

Step 1: Edit the Config File

Step 2: Delete the Existing Kube API Server Cert

Step 3: Restart K3s

Step 4: Inform the DynamicListener About the Change

Alternative Method

Worker / Control Plane Node Reconfiguration (Optional)

Install K3s on Proxmox Using Ansible

Introduction

Table of Contents

Prerequisites

Proxmox Firewall Configuration

Creating Proxmox Security Group

Applying the Security Groups to your VMs

Ansible Setup

Update your SSH config

Directory Structure

Ansible Config File

Ansible Inventory

The Enrollment Token

The K3s Ansible Role

The Playbook

Run the Playbook

Cluster Access

Resources