DEV Community

loading...

ETCD Cluster and Non-voting Learners

Tunde Oladipupo
Open Source Community Advocate. All things Performance and Automation.
・3 min read

In 3.4, etcd introduced raft learner for those looking to add new members to the cluster without giving them voting right. These are particularly useful because we can add members to the cluster without causing disruption then promote them to voting members once everything is good to go. Check this out

Steps to add a learner to the cluster;

  • Add the learner to existing cluster as below;
etcdctl member add etcd-node-3 --peer-urls="http://192.168.13.10:2380" --learner
Member 14d2e95e16c995c7 added to cluster 7ef1685daf3d8f18

ETCD_NAME="etcd-node-3"
ETCD_INITIAL_CLUSTER="etcd-node-3=http://192.168.13.10:2380,etcd-node-1=http://192.168.11.10:2380,etcd-node-0=http://192.168.10.10:2380,etcd-node-2=http://192.168.12.10:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.13.10:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

Once add, you can check this status;

# etcdctl member list -w table 
+------------------+---------+-------------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |    NAME     |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------------+---------------------------+---------------------------+------------+
| 14d2e95e16c995c7 | started | etcd-node-3 | http://192.168.13.10:2380 | http://192.168.13.10:2379 |       true |
| 4c3c38d6652b5d75 | started | etcd-node-1 | http://192.168.11.10:2380 | http://192.168.11.10:2379 |      false |
| a00eaaf5194c573d | started | etcd-node-0 | http://192.168.10.10:2380 | http://192.168.10.10:2379 |      false |
| fdd13ca43538c5a2 | started | etcd-node-2 | http://192.168.12.10:2380 | http://192.168.12.10:2379 |      false |
+------------------+---------+-------------+---------------------------+---------------------------+------------+

You can see our new member is a learner, meaning it syncs with the cluster but does not have voting right or accept read or write request but it is not started yet.

etcdctl member list
16f23207f1b6cfc0, unstarted, , http://192.168.13.10:2379, , true
4c3c38d6652b5d75, started, etcd-node-1, http://192.168.11.10:2380, http://192.168.11.10:2379, false
a00eaaf5194c573d, started, etcd-node-0, http://192.168.10.10:2380, http://192.168.10.10:2379, false
fdd13ca43538c5a2, started, etcd-node-2, http://192.168.12.10:2380, http://192.168.12.10:2379, false

You can now start the learner with the outputted configuration.

 /usr/local/bin/etcd \
  --data-dir=/etcd-data --name etcd-node-3 \
  --initial-advertise-peer-urls http://192.168.13.10}:2380 --listen-peer-urls http://0.0.0.0:2380 \
  --advertise-client-urls http://192.168.13.10:2379 --listen-client-urls http://0.0.0.0:2379 \
  --initial-cluster "etcd-node-3=http://192.168.13.10:2380,etcd-node-1=http://192.168.11.10:2380,etcd-node-0=http://192.168.10.10:2380,etcd-node-2=http://192.168.12.10:2380" \
  --initial-cluster-state existing \
  --initial-cluster-token my-etcd-token

Now, it should show the learner as started;

# etcdctl member list -w table
+------------------+---------+-------------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |    NAME     |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------------+---------------------------+---------------------------+------------+
| 14d2e95e16c995c7 | started | etcd-node-3 | http://192.168.13.10:2380 | http://192.168.13.10:2379 |       true |
| 4c3c38d6652b5d75 | started | etcd-node-1 | http://192.168.11.10:2380 | http://192.168.11.10:2379 |      false |
| a00eaaf5194c573d | started | etcd-node-0 | http://192.168.10.10:2380 | http://192.168.10.10:2379 |      false |
| fdd13ca43538c5a2 | started | etcd-node-2 | http://192.168.12.10:2380 | http://192.168.12.10:2379 |      false |
+------------------+---------+-------------+---------------------------+---------------------------+------------+

This is where learner is of importance; if any of our voting members goes down and our cluster is in inconsistent quorum state. You can promote the learner to replace the unstable voting member.

etcdctl member promote 14d2e95e16c995c7
Member 14d2e95e16c995c7 promoted in cluster 7ef1685daf3d8f18

Discussion (0)