DEV Community 👩‍💻👨‍💻

Cover image for Customized Kubespray Deployment
Chuck Ha
Chuck Ha

Posted on

Customized Kubespray Deployment

Versions

kubespray ansible python terraform
this fork 2.7.0 3.6.1 0.11.8

⚠️ Warning! Use this fork: https://github.com/kubernetes-incubator/kubespray/pull/3486.

Motivation

This post serves as documentation for creating a kubernetes cluster from nothing using kubespray on aws with ubuntu images behind a bastion host. I could not find all the documentation that puts all of this together and wanted to write it down for myself the next time I need to do this.

Steps

  1. Clone kubespray and set up some default files
git clone https://github.com/bartlaarhoven/kubespray
cd kubespray
virtualenv ks && . ks/bin/activate
pip install -r requirements.txt
cp -Rp inventory/sample/ inventory/mycluster
Enter fullscreen mode Exit fullscreen mode
  1. Create an IAM user with admin privileges in some account (TODO probably scope this down?)
  2. Create an EC2 key pair
  3. Copy the terraform environment file to credentials.tfvars and modify it with the user's key and secret along with the ssh key pair name and the region you'd like the infrastructure to exist in.
  4. Customize the terraform file with the architecture you'd like, I used 1 master, 1 worker, 1 etcd and left bastions as default. Also modify the inventory file to be ../../../inventory/mycluster/hosts.ini.
  5. Modify the variables.tf to be
    data "aws_ami" "distro" {
      most_recent = true

      filter {
        name = "name"

        values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-*"]
      }

      filter {
        name = "virtualization-type"

        values = ["hvm"]
      }

      owners = ["099720109477"]
    }
Enter fullscreen mode Exit fullscreen mode
  1. Run terraform
    terraform apply --var-file credentials.tfvars
Enter fullscreen mode Exit fullscreen mode
  1. Modify the ansible.cfg file to use a bastion host by changing the ssh_args value to
    ssh_args = -F ssh-bastion.conf
Enter fullscreen mode Exit fullscreen mode
  1. Modify the hosts.ini file. Use the internal DNS names as the ansible_host for each of the instances in the private subnet and include the ansible_user to be ubuntu, for example:
    kubernetes-devtest-master0 ansible_host=ip-10-250-205-127.us-west-2.compute.internal ansible_user=ubuntu
Enter fullscreen mode Exit fullscreen mode
  1. Modify the bastion lines to be the public DNS names as the ansible_host and include the ansible_user to be ubuntu, for example:
    bastion-0 ansible_host=ec2-22-222-22-22.us-east-2.compute.amazonaws.com ansible_user=ubuntu
Enter fullscreen mode Exit fullscreen mode
  1. Run ansible-playbook
    ansible-playbook -i ./inventory/mycluster/hosts.ini ./cluster.yml -b --become-user=root --flush-cache
Enter fullscreen mode Exit fullscreen mode

If you'd like more logs add -v or -vv up to -vvvvv. I also like to pipe this to tee and write the logs to disk or inspection later in case of failure.

Enjoy your new cluster~!

P.S. if anyone knows how to get the code as part of the item so the bulleted stuff works please comment, I'd love to fix the numbering.

Top comments (2)

Collapse
joehobot profile image
Joe Hobot

I find kubespray more for like baremetal and yeah its nice to have it for aws, however wouldn't something like KOPS be better to standup a cluster?

Collapse
chuck_ha profile image
Chuck Ha Author

Yes, probably! I was trying to reproduce a very specific bug which is why I was using this particular architecture

🌚 Browsing with dark mode makes you a better developer.

It's a scientific fact.