Multi AZ Kubernetes with kops in AWS regions with only 2 AZ

At the moment we are setting up Kubernetes at store2be. We are a company from Germany and most of our business is taking place in Germany. Therefore, we want our AWS servers to be in Frankfurt.

We use kops for our Kubernetes cluster management and we want our cluster to be a multi availability zone (AZ) deployment. With this in mind we tried out the following command:

$ kops create cluster \
    --node-count 4 \
    --zones eu-central-1a,eu-central-1b \
    --master-zones eu-central-1a,eu-central-1b \
    --node-size t2.medium \
    --master-size m3.medium \
    --yes \
    kubernetes-cluster.example.org

But the result was not that promising:

There should be an odd number of master-zones, for etcd's quorum.  Hint: Use --zones and --master-zones to declare node zones and master zones separately.

So unfortunately, at the moment it seems that it is not possible to use kops out of the box in an AWS region with only 2 AZ. But after some digging we found a Github issue that described our problem and even had a work around for that! So the following steps are based on the solution provided by Kamil Hristov.

1. Create a cluster.yml for a AWS region with 3 AZ

Create a cluster in eu-west (because it has 3 AZ):

$ kops create cluster \
    --node-count 4 \
    --zones eu-west-1a,eu-west-1b,eu-west-1c \
    --master-zones eu-west-1a,eu-west-1b,eu-west-1c \
    --node-size t2.medium \
    --master-size m3.medium \
    --yes \
    kubernetes-cluster.example.org

Dump the cluster and instancegroup configuration and append it:

$ kops get cluster --name=kubernetes-cluster.example.org --output yaml > cluster.yml && \
  echo "\n---\n" >> cluster.yml && \
  kops get instancegroups --name=kubernetes-cluster.example.org --output yaml >> cluster.yml

`

2. Adjust the cluster.yml

• Replace all the occurrences of eu-west with eu-central
• Replace all the zone definitions of eu-central-1c with eu-central-1a
• Replace all the name definitions of eu-central-1c with something different than eu-central-1a like for example eu-central-1a2 (otherwise there will be name conflicts)

3. Create the cluster by using the adjusted cluster.yml

`

We have to use replace here, because the cluster spec already has

been created in step 1.

$ kops replace -f cluster.yml
$ kops create secret --name kubernetes-cluster.example.org sshpublickey admin -i ~/.ssh/id_rsa.pub
$ kops update cluster --yes
`

And ✨, there is the multi AZ cluster in Frankfurt 🎉

I created this article because it took us a while to find out about this solution and I hope we could help someone with it. Thanks to Kamil Hristov for sharing the solution. Thanks to Tom Houlé for reviewing the article.

---

Hi there, we're store2be, a Berlin based startup that builds a SaaS enabled marketplace for short term retails space. If you like what we are posting you might wanna check out the store2be tech page or follow our Medium channel.

Comments

[Yevgeniy Ovsyannikov]

This should be fixed!

kops create cluster --zones=us-east-1c --master-count=3 k8s.example.com will create 3 masters all in us-east-1c

You can also:

kops create cluster --zones=us-east-1b,us-east-1c --master-zones=us-east-1b,us-east-1c --master-count=3, it will round-robin around the master zones, so it will so pick b,c,b

[Janusz Bialy]

In terms of high availability and in order to maintain proper quorum for the cluster, an odd number of master nodes should be used!

github.com/kubernetes/kops/blob/ma...

[Peter Gundel]

This is true and this article doesn't state otherwise. It's just about getting a kind of high availability if you only have two AZ available.

[fii]

How is storage shared between the AZs ? is does your storage span all AZs ?

[Peter Gundel]

We don't use storage yet, because we only have containers that only need databases/S3 so far. But as far as I know this is not possible in AWS because EBSs can only be attached to EC2 instances in the same AZ.