DEV Community

Peter Gundel
Peter Gundel

Posted on

Multi AZ Kubernetes with kops in AWS regions with only 2 AZ

At the moment we are setting up Kubernetes at store2be. We are a company from Germany and most of our business is taking place in Germany. Therefore, we want our AWS servers to be in Frankfurt.

We use kops for our Kubernetes cluster management and we want our cluster to be a multi availability zone (AZ) deployment. With this in mind we tried out the following command:

$ kops create cluster \
    --node-count 4 \
    --zones eu-central-1a,eu-central-1b \
    --master-zones eu-central-1a,eu-central-1b \
    --node-size t2.medium \
    --master-size m3.medium \
    --yes \
    kubernetes-cluster.example.org
Enter fullscreen mode Exit fullscreen mode

But the result was not that promising:

There should be an odd number of master-zones, for etcd's quorum.  Hint: Use --zones and --master-zones to declare node zones and master zones separately.
Enter fullscreen mode Exit fullscreen mode

So unfortunately, at the moment it seems that it is not possible to use kops out of the box in an AWS region with only 2 AZ. But after some digging we found a Github issue that described our problem and even had a work around for that! So the following steps are based on the solution provided by Kamil Hristov.

1. Create a cluster.yml for a AWS region with 3 AZ

Create a cluster in eu-west (because it has 3 AZ):

$ kops create cluster \
    --node-count 4 \
    --zones eu-west-1a,eu-west-1b,eu-west-1c \
    --master-zones eu-west-1a,eu-west-1b,eu-west-1c \
    --node-size t2.medium \
    --master-size m3.medium \
    --yes \
    kubernetes-cluster.example.org
Enter fullscreen mode Exit fullscreen mode

Dump the cluster and instancegroup configuration and append it:

$ kops get cluster --name=kubernetes-cluster.example.org --output yaml > cluster.yml && \
  echo "\n---\n" >> cluster.yml && \
  kops get instancegroups --name=kubernetes-cluster.example.org --output yaml >> cluster.yml
Enter fullscreen mode Exit fullscreen mode


`

2. Adjust the cluster.yml

  • Replace all the occurrences of eu-west with eu-central
  • Replace all the zone definitions of eu-central-1c with eu-central-1a
  • Replace all the name definitions of eu-central-1c with something different than eu-central-1a like for example eu-central-1a2 (otherwise there will be name conflicts)

3. Create the cluster by using the adjusted cluster.yml

`

We have to use replace here, because the cluster spec already has

been created in step 1.

$ kops replace -f cluster.yml
$ kops create secret --name kubernetes-cluster.example.org sshpublickey admin -i ~/.ssh/id_rsa.pub
$ kops update cluster --yes
`

And ✨, there is the multi AZ cluster in Frankfurt 🎉

I created this article because it took us a while to find out about this solution and I hope we could help someone with it. Thanks to Kamil Hristov for sharing the solution. Thanks to Tom Houlé for reviewing the article.


Hi there, we're store2be, a Berlin based startup that builds a SaaS enabled marketplace for short term retails space. If you like what we are posting you might wanna check out the store2be tech page or follow our Medium channel.

Top comments (5)

Collapse
 
yevgeniyo profile image
Yevgeniy Ovsyannikov

This should be fixed!

kops create cluster --zones=us-east-1c --master-count=3 k8s.example.com will create 3 masters all in us-east-1c

You can also:

kops create cluster --zones=us-east-1b,us-east-1c --master-zones=us-east-1b,us-east-1c --master-count=3, it will round-robin around the master zones, so it will so pick b,c,b

Collapse
 
jbialy profile image
Janusz Bialy

In terms of high availability and in order to maintain proper quorum for the cluster, an odd number of master nodes should be used!

github.com/kubernetes/kops/blob/ma...

Collapse
 
peterfication profile image
Peter Gundel

This is true and this article doesn't state otherwise. It's just about getting a kind of high availability if you only have two AZ available.

Collapse
 
fii profile image
fii

How is storage shared between the AZs ? is does your storage span all AZs ?

Collapse
 
peterfication profile image
Peter Gundel

We don't use storage yet, because we only have containers that only need databases/S3 so far. But as far as I know this is not possible in AWS because EBSs can only be attached to EC2 instances in the same AZ.