DEV Community

Cover image for Kubernetes ETCD Backup And Restore
Adeoye Malumi
Adeoye Malumi

Posted on

Kubernetes ETCD Backup And Restore

#kubernetes #devops #learning #cloudcomputing

Introduction

If you are learning Kubernetes, you will eventually hear a simple but important statement:

etcd is the database of a Kubernetes cluster.

That description is short, but it has a major consequence: if the etcd data is lost or corrupted, the Kubernetes control plane can lose the information it needs to know what exists in the cluster.

In this lab, I practiced an end-to-end etcd backup and restore workflow on a single-node kubeadm cluster running on a machine named osboxes.

The exercise covered:

  • identifying the etcd pod and confirming that the control plane was healthy;
  • locating the static etcd manifest;
  • inspecting the etcd configuration;
  • installing etcdctl and etcdutl;
  • creating an etcd snapshot;
  • validating the snapshot;
  • deliberately creating an nginx Deployment so there was something meaningful to restore;
  • deleting the Deployment;
  • restoring the etcd snapshot into a separate data directory;
  • changing the static etcd manifest to use the restored data;
  • troubleshooting an etcd pod that remained Pending;
  • restarting kubelet;
  • confirming that etcd eventually returned to Running.

The important part of this exercise was not simply getting a successful backup command. It was understanding what each error meant and how to reason from the evidence in the terminal.

The terminal session showed the etcd pod running before the backup, with the control plane components also healthy. Later, after changing the etcd static pod configuration, etcd became Pending; the investigation eventually led to restarting kubelet, after which etcd became 1/1 Running. The original terminal output is the source for the commands and results reproduced throughout this article.

1. First: What is etcd?

Before touching commands, it helps to understand what we are backing up.

Kubernetes has many components, but the important relationship for this exercise is roughly:

kubectl
   |
   v
kube-apiserver
   |
   v
  etcd
   |
   +---- Kubernetes cluster state
Enter fullscreen mode Exit fullscreen mode

The API server is the main interface through which Kubernetes clients interact with the cluster. etcd is the persistent key-value store used by Kubernetes to store cluster state.

That state includes information such as:

  • Kubernetes objects;
  • Deployments;
  • Pods and their specifications;
  • Services;
  • ConfigMaps;
  • Secrets;
  • cluster configuration and metadata.

A useful mental model for a beginner is:

The Kubernetes API is how you ask Kubernetes about the cluster; etcd is where much of that cluster state is persisted.

This is why backing up etcd is fundamentally different from simply exporting Kubernetes YAML.

2. Verify that the cluster is healthy first

Before taking a backup, I first checked the cluster.

Terminal 

┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A
NAMESPACE     NAME                                       READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-74c68c8864-fjtnv   0/1     ContainerCreating   0          3m10s
kube-system   calico-node-dzlcp                          0/1     Init:2/3             0          3m10s
kube-system   coredns-589f44dc88-l522w                   0/1     ContainerCreating   0          8m56s
kube-system   coredns-589f44dc88-nzsh9                   0/1     ContainerCreating   0          8m56s
kube-system   etcd-osboxes                               1/1     Running             0          9m6s
kube-system   kube-apiserver-osboxes                     1/1     Running             0          9m9s
kube-system   kube-controller-manager-osboxes            1/1     Running             0          9m5s
kube-system   kube-proxy-d8krd                           1/1     Running             0          8m56s
kube-system   kube-scheduler-osboxes                     1/1     Running             0          9m8s
Enter fullscreen mode Exit fullscreen mode

The most important observation here is:

etcd-osboxes  1/1  Running
Enter fullscreen mode Exit fullscreen mode

The API server, controller manager, scheduler and kube-proxy were also running.

Some networking components were still starting, so I watched the pods:

┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A -w
Enter fullscreen mode Exit fullscreen mode

Eventually CoreDNS and Calico became ready.

I then checked the node:

┌──(osboxes㉿osboxes)-[~]
└─$ k get nodes
NAME      STATUS   ROLES           AGE   VERSION
osboxes   Ready    control-plane   10m   v1.36.3
Enter fullscreen mode Exit fullscreen mode

At this point the single control-plane node was Ready.

The initial cluster state is important because a backup should be taken from a functioning etcd instance rather than blindly assuming that etcd is healthy.

3. An important distinction: Kubernetes YAML backup vs etcd backup

Before working directly with etcd, I also ran:

┌──(osboxes㉿osboxes)-[~]
└─$ k get all -A -o yaml > backup.yaml
Enter fullscreen mode Exit fullscreen mode

This produced:

┌──(osboxes㉿osboxes)-[~]
└─$ ls
backup.yaml  Documents  kubectl  Pictures  Public  Videos
Desktop      Downloads  Music    Projects  Templates  VM-Share
Enter fullscreen mode Exit fullscreen mode

This is useful, but it is not the same thing as an etcd snapshot.

The command:

k get all -A -o yaml > backup.yaml
Enter fullscreen mode Exit fullscreen mode

exports Kubernetes resources returned by kubectl get all.

An etcd snapshot, on the other hand, is a snapshot of the etcd backend itself.

For an etcd disaster-recovery exercise, the etcd snapshot is the important artifact.

4. Finding the etcd static pod

This cluster was created with kubeadm, so etcd was running as a static pod.

I initially made a typo:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kkubenetes/manifests
bash: cd: /etc/kkubenetes/manifests: No such file or directory
Enter fullscreen mode Exit fullscreen mode

The problem was simply the spelling:

kkubenetes
Enter fullscreen mode Exit fullscreen mode

instead of:

kubernetes
Enter fullscreen mode Exit fullscreen mode

I then made another typo:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubenetes/manifests
bash: cd: /etc/kubenetes/manifests: No such file or directory
Enter fullscreen mode Exit fullscreen mode

Again, the correct directory is:

/etc/kubernetes/manifests
Enter fullscreen mode Exit fullscreen mode

Eventually:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubernetes/manifests
Enter fullscreen mode Exit fullscreen mode

and:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml
Enter fullscreen mode Exit fullscreen mode

This directory is extremely important on a kubeadm control-plane node.

The files in /etc/kubernetes/manifests are static pod manifests. The kubelet watches this directory and makes sure the described control-plane containers are running.

That means editing etcd.yaml is effectively changing how the control-plane etcd process is launched.

5. Another wrong assumption: trying to use Docker

I also tried:

┌──(osboxes㉿osboxes)-[~]
└─$ docker exec -it osboxes bash
Error response from daemon: No such container: osboxes
Enter fullscreen mode Exit fullscreen mode

This failed because the Kubernetes node name osboxes is not necessarily a Docker container name.

In this setup, the node was running Kubernetes with containerd, and the etcd pod later showed a container ID beginning with:

containerd://
Enter fullscreen mode Exit fullscreen mode

So the important lesson is:

Do not assume that a Kubernetes node or Pod name is a Docker container name.

If you need to inspect a Kubernetes container, first determine which container runtime the node is using.

6. Inspecting the etcd manifest

The first attempt to read the manifest without elevated privileges also failed:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cat etcd.yaml
cat: etcd.yaml: Permission denied
Enter fullscreen mode Exit fullscreen mode

The solution was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo cat etcd.yaml
Enter fullscreen mode Exit fullscreen mode

The manifest revealed several important values.

For example:

- --advertise-client-urls=https://10.0.2.15:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls=https://127.0.0.1:2379,https://10.0.2.15:2379
Enter fullscreen mode Exit fullscreen mode

It also showed that the etcd data directory was originally:

/var/lib/etcd
Enter fullscreen mode Exit fullscreen mode

and the TLS certificates were under:

/etc/kubernetes/pki/etcd/
Enter fullscreen mode Exit fullscreen mode

This information is essential because etcdctl needs to connect to the correct endpoint and authenticate with the appropriate certificates.

7. Install etcdctl

The machine did not initially have the etcd client installed, so I installed it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo apt install etcd-client
Enter fullscreen mode Exit fullscreen mode

The package installed successfully:

Installing:
  etcd-client

Summary:
  Upgrading: 0, Installing: 1, Removing: 0, Not Upgrading: 205
Enter fullscreen mode Exit fullscreen mode

I then checked the snapshot command:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ETCDCTL_API=3 etcdctl snapshot
Enter fullscreen mode Exit fullscreen mode

The output showed:

COMMANDS:
    restore  Restores an etcd member snapshot to an etcd directory
    save     Stores an etcd node backend snapshot to a given file
    status   [deprecated] Gets backend snapshot status of a given file
Enter fullscreen mode Exit fullscreen mode

The key command for the backup is:

etcdctl snapshot save
Enter fullscreen mode Exit fullscreen mode

8. ETCDCTL_API=3

I also explicitly exported:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ export ETCDCTL_API=3
Enter fullscreen mode Exit fullscreen mode

For the lab, this made it explicit that etcdctl should use the v3 API.

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl snapshot
Enter fullscreen mode Exit fullscreen mode

again showed the snapshot subcommands.

For a beginner, the important idea is:

ETCDCTL_API=3
Enter fullscreen mode Exit fullscreen mode

selects the etcd API version used by the client.

9. Creating the first etcd snapshot — and making a typo

I attempted the backup with:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379 \
> --cacert=/etc/kubernetes/pki/etcd/ca.crt \
> --cert=/etc/kubernetes/pki/etcd/server.crt \
> --key=/etc/kubernetes/pki/etcd/server.key \
> snapshot save /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

The error was:

Error: bad flag syntax: ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379
Enter fullscreen mode Exit fullscreen mode

The mistake was the number of hyphens.

I typed:

---endpoint
Enter fullscreen mode Exit fullscreen mode

instead of:

--endpoints
Enter fullscreen mode Exit fullscreen mode

There were actually two problems worth noticing:

  1. There were three hyphens.
  2. The correct flag is --endpoints, plural.

This is a classic CLI problem: the command can look almost correct to a human while being completely invalid to the parser.

10. The next error: permission denied

I corrected the flag and tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

The command reached the certificate files but failed with:

Error: open /etc/kubernetes/pki/etcd/server.key: permission denied
Enter fullscreen mode Exit fullscreen mode

This made sense.

The etcd private key is protected, and the regular user did not have permission to read it.

The solution was to run the command with sudo:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

This time the backup succeeded.

11. The successful etcd backup

The successful command produced:

{"level":"info","ts":"2026-08-13T12:11:59.649233-0400","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/opt/etcd-backup.db.part"}
{"level":"info","ts":"2026-08-13T12:11:59.660384-0400","logger":"client","caller":"v3/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-08-13T12:11:59.660467-0400","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2026-08-13T12:11:59.831074-0400","logger":"client","caller":"v3/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-08-13T12:11:59.894020-0400","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.9 MB","took":"now"}
{"level":"info","ts":"2026-08-13T12:11:59.894158-0400","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/opt/etcd-backup.db"}
Snapshot saved at /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

The resulting backup file was:

/opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

The snapshot was approximately:

3.9 MB
Enter fullscreen mode Exit fullscreen mode

This was the first major milestone.

12. Validating the snapshot

I initially tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --write-out=table snapshot status snapshot.db
Enter fullscreen mode Exit fullscreen mode

The client responded:

Deprecated: Use `etcdutl snapshot status` instead.

Error: stat snapshot.db: no such file or directory
Enter fullscreen mode Exit fullscreen mode

There were two lessons here.

Lesson 1: The filename was wrong

The backup was saved as:

/opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

not:

snapshot.db
Enter fullscreen mode Exit fullscreen mode

Lesson 2: The command was deprecated

The output explicitly recommended:

etcdutl snapshot status
Enter fullscreen mode Exit fullscreen mode

So I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db
Enter fullscreen mode Exit fullscreen mode

but etcdutl was not installed.

The system suggested:

sudo apt install etcd-server
Enter fullscreen mode Exit fullscreen mode

I installed the package.

After installation, I still used the wrong filename once:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db
Error: stat snapshot.db: no such file or directory
Enter fullscreen mode Exit fullscreen mode

Then I used the correct path:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

This produced:

Error: open /opt/etcd-backup.db: permission denied
Enter fullscreen mode Exit fullscreen mode

Again, the issue was file permissions.

So I used sudo:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

The result was:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ a633855e │     5973 │       1298 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘
Enter fullscreen mode Exit fullscreen mode

This confirmed that the snapshot was readable and contained etcd data.

13. Why create an nginx Deployment?

A backup is much more meaningful if we have something we can verify after restoring it.

At this point, I created an nginx Deployment.

My first attempt was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx
error: required flag(s) "image" not set
Enter fullscreen mode Exit fullscreen mode

The command was missing the container image.

I then tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy --image=nginx
error: exactly one NAME is required, got 0
See 'kubectl create deployment -h' for help and examples
Enter fullscreen mode Exit fullscreen mode

Now the image was present, but the Deployment name was missing.

The correct command was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx --image=nginx
deployment.apps/nginx created
Enter fullscreen mode Exit fullscreen mode

This is a good beginner lesson:

kubectl create deployment <NAME> --image=<IMAGE>
Enter fullscreen mode Exit fullscreen mode

So:

kubectl create deployment nginx --image=nginx
Enter fullscreen mode Exit fullscreen mode

means:

  • create a Deployment;
  • call it nginx;
  • use the nginx container image.

14. Waiting for the nginx Pod

Immediately after creation:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                        READY   STATUS              RESTARTS   AGE
pod/nginx-7f8fbb96d-9f7wm   0/1     ContainerCreating   0          4s
Enter fullscreen mode Exit fullscreen mode

The Pod remained in:

ContainerCreating
Enter fullscreen mode Exit fullscreen mode

I watched it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -w
Enter fullscreen mode Exit fullscreen mode

After waiting, I checked again.

The Pod eventually became:

nginx-7f8fbb96d-9f7wm   1/1   Running   0   3m16s
Enter fullscreen mode Exit fullscreen mode

The describe output explained why it initially took time:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ---   ----               -------
  Normal  Scheduled  83s   default-scheduler  Successfully assigned default/nginx-7f8fbb96d-9f7wm to osboxes
  Normal  Pulling    82s   kubelet            spec.containers{nginx}: Pulling image "nginx"
Enter fullscreen mode Exit fullscreen mode

The Pod had been scheduled successfully. The kubelet was simply pulling the image.

This is an important troubleshooting principle:

ContainerCreating does not automatically mean something is broken. Check the Pod events before changing anything.

Eventually:

k get po
NAME                    READY   STATUS    RESTARTS   AGE
nginx-7f8fbb96d-9f7wm   1/1     Running   0          3m16s
Enter fullscreen mode Exit fullscreen mode

15. Take another backup after the nginx Deployment exists

This step is important because the first snapshot was taken before the nginx Deployment was created.

After nginx was running, I created another snapshot:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

The snapshot was again:

3.9 MB
Enter fullscreen mode Exit fullscreen mode

and the command ended with:

Snapshot saved at /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

I validated it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

Result:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │     6934 │        918 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘
Enter fullscreen mode Exit fullscreen mode

Notice that the revision changed.

The backup now represented a later state of the cluster.

16. Verify the state we want to recover

At this point:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-7f8fbb96d-9f7wm   1/1     Running   0          4m1s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   76m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1            4m2s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-7f8fbb96d   1         1         1       4m2s
Enter fullscreen mode Exit fullscreen mode

The important object is:

deployment.apps/nginx
Enter fullscreen mode Exit fullscreen mode

It was healthy.

Now the test was simple:

If I delete nginx, can I use the etcd snapshot to recover the cluster state that existed when the snapshot was taken?

17. Delete nginx

I deleted the Deployment:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k delete deploy nginx
deployment.apps "nginx" deleted from default namespace
Enter fullscreen mode Exit fullscreen mode

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   76m
Enter fullscreen mode Exit fullscreen mode

The nginx Deployment was gone.

This gave us the desired recovery scenario:

BEFORE BACKUP
    |
    +-- nginx Deployment exists
    |
    v
TAKE ETCD SNAPSHOT
    |
    v
DELETE nginx
    |
    v
nginx no longer exists
    |
    v
RESTORE SNAPSHOT
    |
    v
expect previous cluster state
Enter fullscreen mode Exit fullscreen mode

18. First restore mistake: wrong snapshot filename

I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --data-dir /opt/etcd/backup snapshot restore snapshot.db
Enter fullscreen mode Exit fullscreen mode

The restore process immediately told me:

Error: open snapshot.db: no such file or directory
Enter fullscreen mode Exit fullscreen mode

Again, the filename was wrong.

The actual snapshot was:

/opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

This is a very common command-line mistake: creating a file with one name and later trying to restore a similarly named file.

The first restore attempt was therefore not an etcd failure. It was simply a path/filename error.

19. Restore the snapshot into a separate data directory

Instead of overwriting the active etcd data directory immediately, I restored the snapshot into a separate directory:

/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

The command used was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot restore /opt/etcd-backup.db \
--data-dir=/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

The output included:

Deprecated: Use `etcdutl snapshot restore` instead.
Enter fullscreen mode Exit fullscreen mode

and then:

restoring snapshot
Enter fullscreen mode Exit fullscreen mode

followed by:

Trimming membership information from the backend...
Enter fullscreen mode Exit fullscreen mode

and:

added member
Enter fullscreen mode Exit fullscreen mode

Finally:

restored snapshot
Enter fullscreen mode Exit fullscreen mode

This was the actual successful restore of the snapshot into the new data directory.

20. Why use a separate restore directory?

The original etcd data directory was:

/var/lib/etcd
Enter fullscreen mode Exit fullscreen mode

The restored copy was:

/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

This distinction is important.

It allows the restored data to be prepared without immediately destroying the existing data directory.

For a beginner, think of it like restoring a database backup into a new folder first:

Original:
    /var/lib/etcd

Restored:
    /var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

Only after the restored data exists do we configure etcd to use it.

21. More path mistakes while inspecting the restored data

I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cd /var/lib/data
bash: cd: /var/lib/data: No such file or directory
Enter fullscreen mode Exit fullscreen mode

Again, the directory did not exist.

The restore directory was:

/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

I also tried:

sudo cd /var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

and got:

sudo: cd: command not found
sudo: "cd" is a shell built-in command, it cannot be run directly.
Enter fullscreen mode Exit fullscreen mode

This error is worth understanding.

cd is a shell built-in command. sudo normally executes an external command, so:

sudo cd ...
Enter fullscreen mode Exit fullscreen mode

does not work.

Instead, because the directory required elevated permissions, I used:

sudo ls /var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

which showed:

member
Enter fullscreen mode Exit fullscreen mode

That confirmed that the restore had created the expected etcd member directory structure.

22. Editing the static etcd manifest

The next step was to configure the static etcd pod to use the restored data directory.

The original manifest used:

--data-dir=/var/lib/etcd
Enter fullscreen mode Exit fullscreen mode

The modified configuration used:

--data-dir=/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

The corresponding hostPath was also changed to:

/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

The resulting Pod description later confirmed:

--data-dir=/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

and:

Path: /var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

This is how the restored database was connected to the etcd process.

23. Why moving the manifest files caused disruption

During the troubleshooting process, I temporarily moved the static pod manifests out of the manifest directory:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv * /tmp
Enter fullscreen mode Exit fullscreen mode

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 0
Enter fullscreen mode Exit fullscreen mode

The directory was empty.

I then restored the manifests:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv /tmp/*.yaml .
Enter fullscreen mode Exit fullscreen mode

and confirmed:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 16
-rw------- 1 root root 3939 Aug 13 11:07 kube-apiserver.yaml
-rw------- 1 root root 3230 Aug 13 11:07 kube-controller-manager.yaml
-rw------- 1 root root 1726 Aug 13 11:07 kube-scheduler.yaml
-rw------- 1 root root 2596 Aug 13 12:37 etcd.yaml
Enter fullscreen mode Exit fullscreen mode

This demonstrates how sensitive the static pod manifest directory is.

When the kubelet stops seeing a static pod manifest, the corresponding static pod can disappear.

For this reason, moving files in /etc/kubernetes/manifests should be done carefully.

24. Another small typo

I accidentally ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get oi
error: the server doesn't have a resource type "oi"
Enter fullscreen mode Exit fullscreen mode

The intended command was clearly not oi.

The correct command for Pods was:

k get po
Enter fullscreen mode Exit fullscreen mode

This is a harmless error, but it illustrates another important habit:

When kubectl reports that a resource type does not exist, first check the command spelling before assuming the Kubernetes API is broken.

25. The RBAC error after the manifest changes

After the static pod manipulation, this happened:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po
Error from server (Forbidden): pods is forbidden: User "kubernetes-admin" cannot list resource "pods" in API group "" in the namespace "default"
Enter fullscreen mode Exit fullscreen mode

This was different from the earlier errors.

This time, the Kubernetes API server was responding, but the authenticated user was being denied access.

The command:

sudo k get po
Enter fullscreen mode Exit fullscreen mode

was also not a solution:

sudo: k: command not found
Enter fullscreen mode Exit fullscreen mode

The reason is that k was a shell alias, and aliases normally are not available to sudo in that way.

More importantly, sudo does not automatically solve Kubernetes RBAC problems.

I then checked the system namespace explicitly:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system
Enter fullscreen mode Exit fullscreen mode

This returned the control-plane Pods.

That helped separate the problem from the actual etcd state.

26. etcd was now Pending

The output showed:

etcd-osboxes   0/1   Pending
Enter fullscreen mode Exit fullscreen mode

I first tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po etcd-osboxes
Error from server (NotFound): pods "etcd-osboxes" not found
Enter fullscreen mode Exit fullscreen mode

Again, the namespace mattered.

The correct command was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes
Enter fullscreen mode Exit fullscreen mode

The description showed the important configuration:

Status: Pending
Enter fullscreen mode Exit fullscreen mode

but also:

Image: registry.k8s.io/etcd:3.6.8-0
Enter fullscreen mode Exit fullscreen mode

and:

--data-dir=/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

The Pod also showed the restored directory mounted:

/var/lib/etcd-restore-from-backup from etcd-data
Enter fullscreen mode Exit fullscreen mode

and:

Path: /var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

The kubelet events showed:

Normal  Pulled   ...
Normal  Created  ...
Normal  Started  ...
Enter fullscreen mode Exit fullscreen mode

So the container had actually been created and started even though the Kubernetes status remained Pending.

This is an important troubleshooting clue:

Do not rely on one status field alone. Read the full Pod description and events.

27. Trying to read etcd logs

I tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs etcd-osboxes -n kube-system
Error from server (BadRequest): container "etcd" in pod "etcd-osboxes" is not available
Enter fullscreen mode Exit fullscreen mode

I also accidentally ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs
Enter fullscreen mode Exit fullscreen mode

which returned:

error: expected 'logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER]'.
POD or TYPE/NAME is a required argument for the logs command
Enter fullscreen mode Exit fullscreen mode

The second error was simply because kubectl logs requires a Pod or workload name.

The first error was more interesting: Kubernetes did not consider the container available for normal log retrieval.

I then checked the Pod in wide format:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get pod etcd-osboxes -n kube-system -o wide
NAME           READY   STATUS    RESTARTS   AGE     IP       NODE      NOMINATED NODE   READINESS GATES
etcd-osboxes   0/1     Pending   0          4m58s   <none>   osboxes   <none>           <none>
Enter fullscreen mode Exit fullscreen mode

The lack of a Pod IP was another indication that the Pod was not fully ready.

28. Inspecting kubelet

Since etcd was a static Pod, the kubelet was a key component to inspect.

I ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl status kubelet --no-pager
Enter fullscreen mode Exit fullscreen mode

The result showed:

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; preset: disabled)
     Active: active (running)
Enter fullscreen mode Exit fullscreen mode

The kubelet was running.

The logs also showed repeated activity around the time of the problem.

At this point, rather than making more changes to the etcd manifest, I restarted kubelet:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl restart kubelet
Enter fullscreen mode Exit fullscreen mode

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode

29. The recovery

After restarting kubelet, I checked the system namespace again:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-74c68c8864-fjtnv   1/1     Running   5 (7m47s ago)   95m
calico-node-dzlcp                          1/1     Running   0               95m
coredns-589f44dc88-l522w                   1/1     Running   0               101m
coredns-589f44dc88-nzsh9                   1/1     Running   0               101m
etcd-osboxes                               1/1     Running   0               6m51s
kube-apiserver-osboxes                     1/1     Running   0               101m
kube-controller-manager-osboxes            1/1     Running   1 (7m32s ago)   101m
kube-proxy-d8krd                           1/1     Running   0               101m
kube-scheduler-osboxes                     1/1     Running   1               101m
Enter fullscreen mode Exit fullscreen mode

This was the desired state.

Most importantly:

etcd-osboxes   1/1   Running
Enter fullscreen mode Exit fullscreen mode

30. Confirming the restored etcd configuration

I then described the etcd Pod again:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes
Enter fullscreen mode Exit fullscreen mode

The important portion was:

Status: Running
IP: 10.0.2.15
Enter fullscreen mode Exit fullscreen mode

The etcd container was:

State: Running
Ready: True
Restart Count: 0
Enter fullscreen mode Exit fullscreen mode

And the command still showed:

--data-dir=/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

The mounted volume was:

/var/lib/etcd-restore-from-backup from etcd-data
Enter fullscreen mode Exit fullscreen mode

with the host path:

/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

This confirmed that etcd was now running using the restored data directory.

31. What happened to the nginx Deployment?

The exercise was designed around the snapshot that was taken while nginx existed.

The important sequence was:

1. Create nginx Deployment
2. Wait for nginx to become Running
3. Take etcd snapshot
4. Delete nginx
5. Restore the snapshot
6. Point etcd at restored data
7. Restart kubelet
8. Confirm etcd is Running
Enter fullscreen mode Exit fullscreen mode

The backup validation showed a snapshot revision of:

6934
Enter fullscreen mode Exit fullscreen mode

with:

918 total keys
3.9 MB total size
Enter fullscreen mode Exit fullscreen mode

The restore operation successfully rebuilt an etcd data directory.

The final terminal state confirmed that etcd was successfully running from:

/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

The terminal session provided here does not contain a final k get all after the successful etcd recovery showing nginx reappearing, so I would not claim that nginx restoration was independently verified from the terminal evidence.

That distinction matters.

The exercise successfully demonstrated:

  • snapshot creation;
  • snapshot validation;
  • snapshot restoration;
  • switching the static etcd Pod to restored data;
  • recovery of the etcd Pod.

The source terminal output does not provide a final explicit post-recovery k get all proving that the nginx Deployment was recreated from the restored snapshot.

32. The complete clean backup command

For this environment, the working backup command was:

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

The important pieces are:

Endpoint 

--endpoints=https://127.0.0.1:2379
Enter fullscreen mode Exit fullscreen mode

This tells etcdctl where etcd is listening for client connections.

CA certificate 

--cacert=/etc/kubernetes/pki/etcd/ca.crt
Enter fullscreen mode Exit fullscreen mode

This allows the client to verify the etcd server certificate.

Client certificate 

--cert=/etc/kubernetes/pki/etcd/server.crt
Enter fullscreen mode Exit fullscreen mode

This identifies the client to the TLS-enabled etcd server.

Private key 

--key=/etc/kubernetes/pki/etcd/server.key
Enter fullscreen mode Exit fullscreen mode

This goes with the client certificate.

Snapshot command 

snapshot save /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

This tells etcdctl to create a backend snapshot at the specified path.

33. The clean snapshot validation command

The session showed that etcdctl snapshot status is deprecated and recommended etcdutl.

The working validation command was:

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

Example output from the session:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │     6934 │        918 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘
Enter fullscreen mode Exit fullscreen mode

This is useful because a backup file existing on disk does not, by itself, prove that it is a valid etcd snapshot.

34. The clean restore command

The restore performed successfully with:

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

The terminal explicitly reported:

Deprecated: Use `etcdutl snapshot restore` instead.
Enter fullscreen mode Exit fullscreen mode

So for a newer workflow, prefer the modern etcdutl snapshot restore command rather than copying the deprecated command blindly.

35. Important troubleshooting lessons

Error 1: Wrong directory 

cd /etc/kkubenetes/manifests
Enter fullscreen mode Exit fullscreen mode

and:

cd /etc/kubenetes/manifests
Enter fullscreen mode Exit fullscreen mode

failed because of spelling mistakes.

Fix 

cd /etc/kubernetes/manifests
Enter fullscreen mode Exit fullscreen mode

Error 2: Docker container not found 

docker exec -it osboxes bash
Enter fullscreen mode Exit fullscreen mode

returned:

No such container: osboxes
Enter fullscreen mode Exit fullscreen mode

Lesson

A Kubernetes node name is not automatically a Docker container name.

Error 3: Permission denied reading etcd files 

cat etcd.yaml
Enter fullscreen mode Exit fullscreen mode

returned:

Permission denied
Enter fullscreen mode Exit fullscreen mode

Fix 

sudo cat etcd.yaml
Enter fullscreen mode Exit fullscreen mode

The same principle applied to the protected etcd private key.

Error 4: Invalid etcdctl flag 

---endpoint=...
Enter fullscreen mode Exit fullscreen mode

returned:

Error: bad flag syntax
Enter fullscreen mode Exit fullscreen mode

Fix

Use the correctly spelled flag:

--endpoints=...
Enter fullscreen mode Exit fullscreen mode

Error 5: Private key permission denied 

Error: open /etc/kubernetes/pki/etcd/server.key: permission denied
Enter fullscreen mode Exit fullscreen mode

Fix

Run the etcd client command with sufficient privileges:

sudo etcdctl ...
Enter fullscreen mode Exit fullscreen mode

Error 6: Wrong snapshot filename 

snapshot.db
Enter fullscreen mode Exit fullscreen mode

did not exist.

The actual file was:

/opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

Lesson

Always verify the exact backup path before restoring.

Error 7: etcdctl snapshot status deprecated

The client reported:

Deprecated: Use `etcdutl snapshot status` instead.
Enter fullscreen mode Exit fullscreen mode

Fix

Install/use etcdutl and run:

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

Error 8: etcdutl not installed

The command:

etcdutl
Enter fullscreen mode Exit fullscreen mode

was initially unavailable.

Fix

The system suggested installing etcd-server, which provided the utility in this environment.

Error 9: sudo cd

This failed:

sudo cd /var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

with:

sudo: cd: command not found
Enter fullscreen mode Exit fullscreen mode

Lesson

cd is a shell built-in.

Use commands such as:

sudo ls /var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

or start a privileged shell if you genuinely need one.

Error 10: Pod name not found

This:

k describe po etcd-osboxes
Enter fullscreen mode Exit fullscreen mode

returned:

pods "etcd-osboxes" not found
Enter fullscreen mode Exit fullscreen mode

because the command defaulted to the default namespace.

Fix 

k describe po -n=kube-system etcd-osboxes
Enter fullscreen mode Exit fullscreen mode

Error 11: kubectl logs without a Pod 

k logs
Enter fullscreen mode Exit fullscreen mode

returned:

POD or TYPE/NAME is a required argument for the logs command
Enter fullscreen mode Exit fullscreen mode

Fix

Provide the resource:

k logs <pod-name>
Enter fullscreen mode Exit fullscreen mode

and specify the namespace when needed.

Error 12: kubectl Forbidden

After the control-plane changes:

Error from server (Forbidden): pods is forbidden:
User "kubernetes-admin" cannot list resource "pods"
Enter fullscreen mode Exit fullscreen mode

This was an authorization/RBAC problem rather than a shell permission problem.

sudo did not solve it because Kubernetes authorization is separate from Linux sudo.

Error 13: etcd remained Pending

The etcd Pod stayed:

0/1 Pending
Enter fullscreen mode Exit fullscreen mode

even though the container had been created and started according to the events.

The investigation included:

k describe po -n=kube-system etcd-osboxes
Enter fullscreen mode Exit fullscreen mode

and:

sudo systemctl status kubelet --no-pager
Enter fullscreen mode Exit fullscreen mode

The practical recovery step that resolved the state in this lab was:

sudo systemctl restart kubelet
Enter fullscreen mode Exit fullscreen mode

After that:

etcd-osboxes   1/1   Running
Enter fullscreen mode Exit fullscreen mode

36. Why static Pods matter in this exercise

The etcd Pod was not created with a normal Deployment such as:

kind: Deployment
Enter fullscreen mode Exit fullscreen mode

Instead, it was defined by:

/etc/kubernetes/manifests/etcd.yaml
Enter fullscreen mode Exit fullscreen mode

That makes it a static Pod.

The kubelet watches the manifest directory and manages the Pod locally.

This explains several things that happened during the exercise:

  • changing etcd.yaml affected etcd;
  • temporarily moving manifests out of the directory affected the control plane;
  • restarting kubelet caused it to re-read and reconcile the manifests;
  • etcd could be controlled without creating a normal Kubernetes Deployment.

For kubeadm clusters, understanding static Pods is essential when troubleshooting the control plane.

37. A safer mental model for etcd restore

Do not think of restore as:

"copy backup file over etcd"
Enter fullscreen mode Exit fullscreen mode

A better mental model is:

ETCD SNAPSHOT
     |
     v
restore snapshot
     |
     v
new etcd data directory
     |
     v
configure etcd to use restored directory
     |
     v
kubelet recreates/restarts static etcd Pod
     |
     v
etcd becomes healthy
     |
     v
Kubernetes API becomes available against restored state
Enter fullscreen mode Exit fullscreen mode

This is why the exercise used:

/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

instead of immediately destroying:

/var/lib/etcd
Enter fullscreen mode Exit fullscreen mode

38. What I would do differently next time

The lab worked, but it also exposed several areas where the workflow could be cleaner.

1. Check the exact paths first

Before typing a long command:

ls -l /etc/kubernetes/pki/etcd/
ls -l /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

This avoids guessing filenames and permissions.

2. Inspect the manifest before changing it

First:

sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.backup
Enter fullscreen mode Exit fullscreen mode

Then edit the manifest.

That gives you a known-good copy if something goes wrong.

3. Prefer the current etcd utilities

The terminal explicitly showed that:

etcdctl snapshot status
Enter fullscreen mode Exit fullscreen mode

is deprecated and recommended:

etcdutl snapshot status
Enter fullscreen mode Exit fullscreen mode

Likewise, the restore output recommended:

etcdutl snapshot restore
Enter fullscreen mode Exit fullscreen mode

4. Avoid unnecessary movement of all manifests

This command:

sudo mv * /tmp
Enter fullscreen mode Exit fullscreen mode

is powerful and potentially disruptive.

Moving all control-plane manifests affects more than etcd.

A safer approach is to modify only the required manifest and keep backups of the original.

5. Verify each stage

After every major step:

k get pods -A
Enter fullscreen mode Exit fullscreen mode

or:

k get pods -n kube-system
Enter fullscreen mode Exit fullscreen mode

and inspect:

k describe pod <pod> -n kube-system
Enter fullscreen mode Exit fullscreen mode

if anything looks abnormal.

39. A compact recovery checklist

For a kubeadm single-control-plane lab similar to this one:

Identify etcd 

k get pods -n kube-system | grep etcd
Enter fullscreen mode Exit fullscreen mode

Inspect the manifest 

cd /etc/kubernetes/manifests
sudo cat etcd.yaml
Enter fullscreen mode Exit fullscreen mode

Identify: 

--data-dir
--listen-client-urls
--cert-file
--key-file
--trusted-ca-file
Enter fullscreen mode Exit fullscreen mode

Create snapshot 

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

Validate snapshot 

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

Restore to a separate directory 

sudo etcdutl snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

Configure etcd to use the restored directory

Update:

--data-dir=/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

and the corresponding hostPath.

Restart/reconcile kubelet if required 

sudo systemctl restart kubelet
Enter fullscreen mode Exit fullscreen mode

Verify 

k get pods -n kube-system
Enter fullscreen mode Exit fullscreen mode

Look for:

etcd-osboxes   1/1   Running
Enter fullscreen mode Exit fullscreen mode

40. What this lab actually taught me

The most valuable part of this exercise was not memorizing one command.

It was learning how the pieces connect:

Kubernetes
    |
    +-- kube-apiserver
    |
    +-- kube-controller-manager
    |
    +-- kube-scheduler
    |
    +-- kubelet
            |
            +-- static Pod manifests
                    |
                    +-- etcd
                            |
                            +-- /var/lib/etcd
Enter fullscreen mode Exit fullscreen mode

The backup process interacted directly with etcd:

etcd
  |
  +-- TLS certificates
  |
  +-- etcdctl
  |
  +-- snapshot
  |
  +-- /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

The restore process then created:

/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

and the static Pod was changed to consume that restored data.

Finally, kubelet reconciled the static Pod and etcd returned to:

1/1 Running
Enter fullscreen mode Exit fullscreen mode

41. Final takeaway

An etcd backup is not just another Kubernetes YAML export.

The core workflow is:

1. Verify etcd is healthy.
2. Find the etcd endpoint and TLS certificates.
3. Take an etcd snapshot.
4. Validate the snapshot.
5. Restore the snapshot into a separate data directory.
6. Configure the etcd static Pod to use the restored data.
7. Let kubelet reconcile the static Pod.
8. Verify etcd and the Kubernetes control plane.
Enter fullscreen mode Exit fullscreen mode

The terminal session also demonstrated something that documentation often hides: the commands do not always work on the first attempt.

There were:

  • spelling mistakes;
  • incorrect flags;
  • permission problems;
  • wrong filenames;
  • deprecated commands;
  • missing utilities;
  • namespace mistakes;
  • RBAC errors;
  • attempts to use sudo where it did not apply;
  • a static etcd Pod that remained Pending;
  • and finally a kubelet restart that brought etcd back to Running.

Those failures were not separate from the learning experience. They were the learning experience.

The most useful troubleshooting habit from this lab is simple:

When Kubernetes gives you an error, do not immediately change random things. Read the error, identify which layer produced it, inspect the relevant component, make the smallest reasonable correction, and verify the result.

That is the difference between memorizing Kubernetes commands and actually learning how the control plane works.

Terminal command reference from the lab

Cluster health 

k get pods -A
k get pods -A -w
k get nodes
k get all
k get all -A
Enter fullscreen mode Exit fullscreen mode

Static Pod manifests 

cd /etc/kubernetes/manifests
ls
sudo cat etcd.yaml
Enter fullscreen mode Exit fullscreen mode

Install tooling 

sudo apt install etcd-client
sudo apt install etcd-server
Enter fullscreen mode Exit fullscreen mode

Configure etcdctl 

export ETCDCTL_API=3
Enter fullscreen mode Exit fullscreen mode

Backup 

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

Validate 

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Enter fullscreen mode Exit fullscreen mode

Test workload 

k create deploy nginx --image=nginx
k get po
k get po -w
k describe po nginx-7f8fbb96d-9f7wm
k get all
Enter fullscreen mode Exit fullscreen mode

Delete workload 

k delete deploy nginx
Enter fullscreen mode Exit fullscreen mode

Restore 

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup
Enter fullscreen mode Exit fullscreen mode

Troubleshoot etcd 

k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes
k get pod etcd-osboxes -n kube-system -o wide
k logs etcd-osboxes -n kube-system
sudo systemctl status kubelet --no-pager
sudo systemctl restart kubelet
Enter fullscreen mode Exit fullscreen mode

Final verification 

k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes
Enter fullscreen mode Exit fullscreen mode

Top comments (0)