DEV Community: pongsatt

Setup your own kubernetes cluster on VMs

pongsatt — Sat, 03 Nov 2018 07:03:04 +0000

My requirement is that I want a server I can can play around. I tried Docker with Jenkins CD and it was awesome. Then Docker swarm and Kubernetes came so I wanted to upgrade my Docker environment to a cluster. The problem is that I've got only a PC and I don't want to pay for my playground environment. Then I found that I can achieve this using a tool named kubeadm and some virtual machines. I would like to share my environment and how to set it up as well as my record for steps I used.

I divided the steps into a series of posts so it's easy and concise enough to follow.

Requirement

A kubernetes cluster based on kubeadm
Support dynamic storage provisioning based on NFS
Support Ingress with automatically acquire let's encrypt certificate
Be able to deploy application using Jenkin CD pipeline for both backend and frontend

You can follow each step in each post below.

Setup detail:

Deploy back-end api application with database to the kubernetes cluster with Jenkins CD

pongsatt — Sat, 03 Nov 2018 07:01:54 +0000

We will create a simple graphql api application that connects to mongodb. Then deploy to kubernetes cluster using Jenkins CD.

In addition, we will create ingress that support https so we can access this application securely.

This post is part of series "Setup your own kubernetes cluster on VMs".

Note:

This post assume that you follow from previous post

Prerequisite:

A running kubernetes cluster with storage-class support
Jenkins server with pipeline support
Docker private repository

Build Application

We will create a project called "simpleapi" which is a graphql with mongodb application using nodejs.

1. Create project

mkdir simpleapi
cd simpleapi
yarn init -y

2. Build graphql api

yarn add graphql-yoga mongodb

Create "db.js" at root folder with content below.

const mongoClient = require('mongodb').MongoClient

module.exports = {
    users: () => execute((coll) => coll.find().toArray()
        .then(results => results.map(idToString))),
    createUser: (name) => execute((coll) => coll.insertOne({ name })
        .then(result => idToString(result.ops[0])))
}

function execute(fn) {
    return mongoClient.connect(process.env.DB_URL || 'mongodb://localhost:27017/test')
        .then(client => {
            const result = fn(client.db().collection('users'))
            client.close()
            return result
        })
}

function idToString(doc) {
    const {_id, ...rest} = doc
    return {_id: _id.toString(), ...rest}
}

This program connects mongo at url "mongodb://localhost:27017/test" or url from environment name "DB_URL" if provided.

It also contains 2 functions for querying user list and create new user. We will use these functions in "index.js" below.

Create "index.js" at root folder with content below.

const { GraphQLServer } = require('graphql-yoga')
const db = require('./db')

const typeDefs = `
  type User {
      _id: ID
      name: String
  }

  type Query {
    users: [User]
  }

  type Mutation {
      createUser(name: String!): User!
  }
`

const resolvers = {
  Query: {
    users: (_, { }) => db.users()
  },
  Mutation: {
    createUser: (_, {name}) => db.createUser(name)
  }
}

const server = new GraphQLServer({ typeDefs, resolvers })
server.start(() => console.log('Server is running on http://localhost:4000'))

In this program, we define graphql schema with function to query user and user creation and start server at port 4000.

3. Test graphql api

Note:

You need mongodb running at "mongodb://localhost:27017" to be able to test

Start server by running node index.js and open url "http://localhost:4000".

Create user.

Query user.

Add Jenkins pipeline to project

The api application pipeline will need Dockerfile, kubernetes and Jenkins pipeline configurations.

1. Dockerfile

Here is the nodejs based Dockerfile configuration.

FROM node:8.11.0-alpine

WORKDIR /usr/src

COPY ./*.js package.json yarn.lock ./
RUN yarn install

EXPOSE 4000
CMD [ "node", "index.js"]

1. Kubernetes for database

Create file "k8s/db.yml" with content below.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: simpleapi-db-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "ssdnfs"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
---

apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    name: simpleapi-db
  name: simpleapi-db
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: simpleapi-db
    spec:
      containers:
      - image: mongo
        name: mongo
        ports:
        - name: mongo
          containerPort: 27017
        volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
      volumes:
        - name: mongo-persistent-storage
          persistentVolumeClaim:
            claimName: simpleapi-db-claim
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: simpleapi-db
  name: simpleapi-db
spec:
  ports:
    - port: 27017
      targetPort: 27017
  selector:
    app: simpleapi-db

This configuration will create Persistent Volume Claim that use storage class "ssdnfs" we created from previous post. We also define mongodb instance and its service.

We can access this db within cluster using "simpleapi-db:27017".

2. Kubernetes for application

Create file "k8s/api.yml" with content.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: simpleapi
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: simpleapi
    spec:
      containers:
        - name: simpleapi
          image: "<imageTag>"
          env:
          - name: DB_URL
            value: "mongodb://simpleapi-db:27017/test"
          ports:
          - name: http
            containerPort: 4000
          readinessProbe:
            httpGet:
              path: /
              port: 4000
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /
              port: 4000
            initialDelaySeconds: 3
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: simpleapi
spec:
  ports:
    - name: http
      port: 80
      targetPort: 4000
  selector:
    app: simpleapi

This config composes of a Deployment and Service. We can access this application through service "simpleapi:80".

We also override database url with environment "DB_URL".

will be replaced in the build process since it's generated later.

3. Kubernetes for ingress

As promise, this api will be able to access through https via ingress configuration (Setup from previous post).

Create file "k8s/ingress.yml".

Notes:

If you apply letsencrypt certificate using http01 protocol, you will need to uncomment/comment annotations.
Replace "simpleapi.yourdomain.com" with your real domain name
You will need to create DNS record for "simpleapi.yourdomain.com" so that it can apply for certificate (See previous post for more detail)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: simpleapi-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"

    # use http01 protocol. uncomment line below to use http01
    # kubernetes.io/tls-acme: "true"

    # use dns01 protocol. comment 2 lines below if use http01
    certmanager.k8s.io/acme-challenge-type: "dns01"
    certmanager.k8s.io/acme-dns01-provider: "aws-route53"

spec:
  tls:
  - hosts:
    - simpleapi.yourdomain.com
    secretName: simpleapi-tls
  rules:
  # The host must be identical to the above one
  - host: simpleapi.yourdomain.com
    http:
      paths:
      - path: /
        backend:
          # The name of your service
          serviceName: simpleapi
          servicePort: 80

4. Jenkinsfile

Create "Jenkinsfile" at the root folder.

Notes:

Replace "192.168.1.105:8082" with your private registry
You need to create "User/password" credential name "myregUserPass" to access your Docker repository in Jenkins

pipeline {
    environment {
        IMAGE_TAG = "${env.BUILD_NUMBER}"
        REGISTRY = "192.168.1.105:8082"
        APP = "simpleapi"
    }
    agent { label 'docker' }

    stages {
        stage('build and push image') {
            steps {
                script {
                    docker.withRegistry("http://${REGISTRY}", 'myregUserPass') {
                        docker.build("${APP}").push("${IMAGE_TAG}")
                    }
                }
            }
        }
        stage('deploy') {
            steps {
                sh("sed -i.bak 's|<imageTag>|${REGISTRY}/${APP}:${IMAGE_TAG}|' ./k8s/api.yml")
                sh("kubectl apply -f k8s/")
            }
        }

    }

}

There are 2 stages in this pipeline.

build and push image: build and push new image to docker private repo
deploy: replace and apply all yml files

At this point, your project folders should look like this.

5. Publish project to git repository

You can push to any git. We will use this to configure Jenkins in next step.

Configure kubernetes for docker private registry

Kubernetes cluster will need to know user and password in order to pull image from it. We can set it up by patching service account that pull the image, in this case, default service account.

Create secret

On your master node.

Notes:

Replace with your private registry server
Replace with your registry data

kubectl create secret docker-registry dockerconfig --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "dockerconfig"}]}'

Create Jenkins Pipeline

Now, we will create new pipeline project in Jenkins called "simpleapi".

Set git repository and credential.

Run build and wait until success.

Run kubectl get secret. If you see "simpleapi-tls", means we can access our application through "https://simpleapi.yourdomain.com".

Summary

This is the last post of this series. We learned how to setup cluster until create front and back end with ingress and dynamic storage. I hope these posts provide some values for you. Thanks.

Deploy static front-end application to kubernetes cluster using Jenkins CD and minio with automatic https cert

pongsatt — Sat, 03 Nov 2018 06:59:21 +0000

After we've setup kubernetes cluster on our VMs, we setup nginx ingress automatically apply for certificate with letsencrypt. This post we will deploy static web application using Jenkins CD and minio.

With Jenkins CD, we can define our build and deployment pipeline, and with the power of kubernetes and docker, our application infrastructure also defined along with our source code.

With minio, a file storage server, we can host static web site as a folder like we do with AWS S3 but within our own infrastructure.

Then with nginx ingress that we setup from previous post, we can point website URL to minio file serving.

Note:

This post assume that you follow this series of posts
If you follow this series, you need to be on master node to follow this post

Prerequisite:

A running kubernetes cluster
helm installed
Jenkins server with pipeline support

Setup minio

Note:

minio requires access key and secret. Please replace "myaccesskey" and "mysecret" with you own values

1. Install minio using helm

This run will install minio server in "minio" namespace

helm install --name minio --namespace minio stable/minio \
--set accessKey=myaccesskey,secretKey=mysecret \
--set service.type=NodePort

2. Install minio web client

Install mc cli using command below (need root password).

wget https://dl.minio.io/client/mc/release/linux-amd64/mc && chmod +x mc && sudo mv mc /usr/local/bin/mc

3. Configure minio pointing to the server

Run kubectl get service -n minio to find NodePort.

Run command to add minio configuration folder.

Note:

Replace with your real master node IP address
Replace with real node port from above step
Replace "myaccesskey" and "mysecret" with your own values

mc config host add myminio http://<Master Node IP>:<NodePort> myaccesskey mysecret

Result should look like this.

4. Create minio folder to host static files

In this example, we will create a bucket name "public" to host all static folders.

mc mb -p myminio/public
mc policy download myminio/public

Run mc ls myminio and you should see an empty bucket.

Setup DNS record

In order to apply for https domain certificate, you need to set your DNS record "simpleapp.yourdomain.com" to your ingress service IP. You can see my example from previous post.

Build a static website

We will use react application as our sample static website. Let's build it.

1. Initialize application

Note: you will need nodejs installed on your machine.

npx create-react-app simpleapp
cd simpleapp
npm start

Open "http://localhost:3000" and you should see.

2. Add kubernetes ingress configuration

Add "ingress.yml" under "k8s" folder at the root folder.

Content:

Note:

Replace "simpleapp.yourdomain.com" with real domain
This example use "dns01" protocol to acquire letsencrypt certificate. You can change it to "http01" by comment/uncomment

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: simpleapp-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"

    # use http01 protocol. uncomment line below to use http01
    # kubernetes.io/tls-acme: "true"

    # use dns01 protocol. comment 2 lines below if use http01
    certmanager.k8s.io/acme-challenge-type: "dns01"
    certmanager.k8s.io/acme-dns01-provider: "aws-route53"

    nginx.ingress.kubernetes.io/rewrite-target: "/public/simpleapp"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^/?$ /public/simpleapp/index.html break;
spec:
  tls:
  - hosts:
    - simpleapp.yourdomain.com
    secretName: simpleapp-tls
  rules:
  # The host must be identical to the above one
  - host: simpleapp.yourdomain.com
    http:
      paths:
      - path: /
        backend:
          # The name of your service
          serviceName: minio
          servicePort: 9000

This configuration will point "simpleapp.yourdomain.com" to a folder under "public" bucket on minio to a folder name "simpleapp". Also support redirect from "/" to "/index.html"

3. Add Jenkinsfile pipeline

Create file name "Jenkinsfile" at the root folder.

Content:

pipeline {
  environment {
    ENV = "minio"
    APP = "simpleapp"
  }
  agent { label 'docker' }

  stages {
    stage('build') {
      steps {
        withDockerContainer('node:8.4.0-alpine') { 
            sh 'yarn install'
            sh 'yarn build'
        }
      }
    }

    stage('publish to minio') {
      steps {
        sh "mc cp -r build/ myminio/public/${APP}"
      }
    }

    stage('deploy ingress to kubernetes') {
        steps {
            sh("kubectl --namespace=${ENV} apply -f k8s/ingress.yml")
        }
    }

  }

}

From the configuration, I have a Jenkins build agent with label "docker" that contains "kubernetes" and "mc" cli.

There are 3 stages.

build: react project to "build" folder
publish to minio: copy folder "build" to minio "public/simpleapp"
deploy ingress to kubernetes: deploy "k8s/ingress.yml" to cluster

4. Publish to git repo

You can publish your project to any git repository.

Prepare Jenkins environment

In this example, your jenkins needs to support below cli.

kubectl : with ".kube" configuration folder that point to our kubernetes server
mc : with ".mc" configuration folder that point to our minio instance

Note:

Install these tools will depend on your jenkins environment

1. kubectl configuration

You can just copy "~/.kube/config" from master node to "/home/jenkins/.kube/config".

2. mc configuration

You can just copy "~/.mc/config.json" from master node to "/home/jenkins/.mc/config.json"

If all the configurations is ok, you should be able to run command below inside Jenkins agent.

Build Jenkins pipeline

In this example, we will create Jenkins pipeline that point to Jenkinsfile in the repository.

1. Create Jenkins pipeline project

Create "simpleapp" project.

Enter your git repo url and credential.

2. Build pipeline

Normally, pipeline build should be automatically triggered when the source code is update. This example will use manual trigger.

Build result.

Test your application

Wait until kubectl get secret -n minio return secret name "simpleapp-tls".

If everything is ok, you should be able to open URL "https://simpleapp.yourdomain.com" and see result as below.

Summary

We build static website with Jenkins CD, then deploy to minio file server and kubernetes with automatic letsencrpyt certified. Next, we will do CD with backend code and kubernetes cluster.

Set up nginx ingress with letsencrypt certificate on VMs or bare metal kubernetes cluster

pongsatt — Sat, 03 Nov 2018 06:58:41 +0000

This post is a guide to setup ingress with cert-manager on kubernetes cluster (VMs based) so that the application deploy on the cluster can apply for https certificate automatically.

When we use kubernetes cluster using cloud providers, they already provide ingress mechanism based on their load balancer technology. If we want this capability on VMs or bare metal based cluster, we need to set it up ourselves.

It is part of series "Setup your own kubernetes cluster on VMs".

Prerequisite:

A running kubernetes cluster
A domain name with you have administration access

Note:

For those who follow along, you need to run everything on master node

Install helm

First of all, we need helm, a cluster package manager, that will help us install ingress and cert-manager.

1. Create service account and its permission

Run command.

echo 'apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system' | kubectl apply -f -

2. Install helm

Run below command to install helm from official bash script.

Note: You will need to enter root password.

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash

3. Initialize helm

We need to initialize helm to install server application on the cluster.

helm init --service-account tiller

4. Check installation

Run helm version.

If things are ok, you should see.

Install Ingress

We will install nginx ingress using host network on all worker nodes (DaemonSet).

See this document to understand more about nginx ingress Bare-metal considerations

1. Install Ingress pod

Install using helm command below.

helm install --name nginx-ingress --namespace nginx-ingress stable/nginx-ingress --set controller.hostNetwork=true --set controller.kind=DaemonSet

Open one of your worker node IP address in the browser. If ingress installation successful, you will see.

2. Configure domain name

Configure your dns to point your test domain name to one of your worker node.

In my example, I point my route53 test dns record to my public IP.

I also need to configure my router to forword port 80 and 443 to one of my worker node. Let's say 192.168.1.110.

3. Deploy test application

Run this command to deploy hello world application to the cluster.

kubectl run hello-world --image=gcr.io/google-samples/node-hello:1.0  --port=8080
kubectl expose deployment hello-world --type=NodePort --name=example-service

Try access this application using node port http://<master or worker ip>:<node port>. You should see.

4. Deploy test ingress

Run command below to install ingress for test application.

Note:

Please replace "testingress.yourdomain.com" with your real domain.

echo 'apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  tls:
  - hosts:
    - testingress.yourdomain.com
    secretName: testingress-tls
  rules:
  - host: testingress.yourdomain.com
    http:
      paths:
      - path: /
        backend:
          # The name of your service
          serviceName: example-service
          servicePort: 8080' | kubectl apply -f -

If everything is ok, open url http://testingress.yourdomain.com and you should see.

Install cert-manager

We will install cert-manager to the cluster then create cluster issuer for letsencrypt. The cluster issuer will automatically detect our ingress with tls configure then try to acquire certificate and store keys in secret name "testingress-tls".

1. Install cert-manager

Run this command using helm.

helm install --name cert-manager --namespace kube-system stable/cert-manager

2. Create cluster issuer (use http01 protocol)

Run this command to create cluster issuer.

Note:

Please replace "yourreal@email.com" to your real email

echo 'apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: default
spec:
  acme:
    email: yourreal@email.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    http01: {}' | kubectl apply -f -

Run below to update cert-manager to use cluster issuer created above.

helm upgrade cert-manager stable/cert-manager --namespace kube-system --set ingressShim.defaultIssuerName=letsencrypt-prod --set ingressShim.defaultIssuerKind=ClusterIssuer

Wait for sometimes maybe an hour. Then run kubectl get secret. If you see secret name "testingress-tls", this mean it works. You should be able to access https://testingress.yourdomain.com successfully.

3. Create cluster issuer (use dns01 protocol)

In case that step 2 with http02 protocol does not work, we will need a more advance protocol "dns01".

"dns01" protocol support many cloud providers, this example will be for route53.

3.1 Import IAM policy

Import this to your IAM user that you have access key.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "route53:GetChange",
            "Resource": "arn:aws:route53:::change/*"
        },
        {
            "Effect": "Allow",
            "Action": "route53:ChangeResourceRecordSets",
            "Resource": "arn:aws:route53:::hostedzone/*"
        },
        {
            "Effect": "Allow",
            "Action": "route53:ListHostedZonesByName",
            "Resource": "*"
        }
    ]
}

3.2 Create AWS secret

Create kubernetes secret using below.

Note:

Replace with your real secret key

kubectl create secret -n kube-system generic aws-secret --from-literal=secret-key=<IAM User Secret Key>

3.3 Create Cluster Issuer (DNS01)

Note:

Replace yourreal@email.com with your real email
Replace and and with your real data

echo 'apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: default
spec:
  acme:
    email: yourreal@email.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    dns01:
      providers:
      - name: aws-route53
        route53:
          hostedZoneID: <Route53 hosted zone ID>
          region: <Route53 region>
          accessKeyID: <IAM User Access key>
          secretAccessKeySecretRef:
            name: aws-secret
            key: secret-key' | kubectl apply -f -

3.4 Update test ingress to use DNS01

Note:

Replace yourdomain.com to your real domain

echo 'apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    certmanager.k8s.io/acme-challenge-type: "dns01"
    certmanager.k8s.io/acme-dns01-provider: "aws-route53"
spec:
  tls:
  - hosts:
    - testingress.yourdomain.com
    secretName: testingress-tls
  rules:
  - host: testingress.yourdomain.com
    http:
      paths:
      - path: /
        backend:
          # The name of your service
          serviceName: example-service
          servicePort: 8080' | kubectl apply -f -

Wait until kubectl get secret return secret name "testingress-tls", this mean it works. You should be able to access https://testingress.yourdomain.com successfully.

Summary

At this point, we have a kubernete cluster that we can deploy application that support https. Next, we will set up a static front-end application in the kubernetes cluster.

Setup dynamic storage class provisioner based on NFS

pongsatt — Sat, 03 Nov 2018 06:57:24 +0000

This post is a guide to setup NFS storage class in your kubernetes cluster virtual machine.

It is part of series "Setup your own kubernetes cluster on VMs".

In a cluster, our application can be run on any host machine so application like database will need to know where to persist the data.

Kubernetes cluster supports this by using Persistent Volume (PV) and Persistent Volume Claim (PVC) which is cluster storage resource. We need to define PV with underlying actual storage then define PVC that use it, then bind PVC to an application.

Kubernetes also support Storage Class in which instead of define PV manually, we can set storage class to application PVC as an annotation. Then, cluster will create a PV that match the storage class for us automatically.

Example:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: db-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "ssdnfs"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

Each cloud provider like AWS or Google has its own support for storage class that bind the actual storage with their existing cloud storage.

Since our kubernetes cluster is on a VMs or bare metal, we don't have this capability out of the box.

In this post, I will guide you to setup storage class based on NFS using a program named "nfs-client-provisioner".

Prerequisite:

A kubernetes cluster on VMs or bare metal with RBAC enabled
A NFS server

We will create a storage class name ssdnfs as a default storage class.
Let's assume that we have NFS server on IP 192.168.1.119 and export path /export/k8sdynamic.

If you follow from previous post, you need to be on the master node.

1. Create storage class

Run command below to define our storage class to the cluster.

echo 'apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ssdnfs
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: k8s/nfs' | kubectl apply -f -

2. Create service account and permission

The provisioner needs permission to monitor and create PV for us so we need a service account with appropriate permission.

Create service account.

echo "apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner" | kubectl apply -f -

Define cluster role.

echo 'kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]' | kubectl apply -f -

Bind cluster to service account.

echo 'kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io' | kubectl apply -f -

3. Deploy provisioner as pod

Run command below to deploy nfs-client-provisioner application to the cluster.

echo 'kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccount: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:v3.1.0-k8s1.11
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s/nfs
            - name: NFS_SERVER
              value: 192.168.1.119 # nodes will need nfs-common to access nfs protocol
            - name: NFS_PATH
              value: /export/k8sdynamic
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.1.119
            path: /export/k8sdynamic' | kubectl apply -f -

Run kubectl get deployment nfs-client-provisioner and wait until you see.

4. Test our storage class

We will create PVC that use our "ssdnfs" storage class and run a pod that use this PVC.

Create PVC.

echo 'kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "ssdnfs"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi' | kubectl apply -f -

Create test pod.

echo 'kind: Pod
apiVersion: v1
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: gcr.io/google_containers/busybox:1.24
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: test-claim' | kubectl apply -f -

Run kubectl get pod test-pod and if you see status "completed", it means our storage class works!

Summary

At this point, we have our own storage class that we can use for our application in our kubernetes cluster. Next, we will setup nginx ingress with cert-manager so we can use https with our application.

Setup kubernetes cluster using kubeadm

pongsatt — Sat, 03 Nov 2018 06:56:11 +0000

From previous post, you prepare a number of VMs machine, in this post, we will install kubeadm and setup it to work together as a cluster.

This is part of Setup your own kubernetes cluster on VMs series.

Note

This is an opinionated kubernetes cluster setup so it is easy enough for beginners. Meaning, something have already been chosen for you but you can change it if you want.
This post is based on this kubernetes official document

Prerequisite:

At least 2 Virtual machines with Ubuntu installed and connected network

Switch to root user (All machines)

Run below command and enter root password (password when you setup VM)

sudo su

Install Docker (All machines)

Follow these steps to install docker to all machines.

1. Install Docker

# Install Docker from Ubuntu's repositories:
apt-get update
apt-get install -y docker.io

2. Configure docker

In case you want to use your own docker private registry, follow below, otherwise follow "Without private registry below"

Note: "192.168.1.105:8082" is my docker private registry.

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",

  "insecure-registries": [
    "192.168.1.105:8082"
  ],
  "disable-legacy-registry": true
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

Without private registry.

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

3. (Optional) Install image cleanup job

I recommend to install this job to clean up unused docker images otherwise they will consume all your disk.

docker run -d --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock:rw \
  -v /var/lib/docker:/var/lib/docker:rw \
  -e "CLEAN_PERIOD=86400" \
  meltwater/docker-cleanup:latest

4. Check if everything alright

Run below command to see if everything is ok.

docker ps

You should see something like below on all machines and you're good to proceed.

Install Kubeadm (All machines)

Next step is to install kubernetes cluster tool.

1. Disable swap

This is a prerequisite for kubeadm.

# disable temporarily
swapoff -a

To disable permanently, edit /etc/fstab and command swap line out and save.

vi /etc/fstab

Note for vi newby:

move cursor to the beginning of last line
type 'i' to insert
type '#' then 'esc'
type ':wq' then enter to save

2. Run below commands on all machines

apt-get update && apt-get install -y apt-transport-https curl

Then

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl nfs-common

And

apt-mark hold kubelet kubeadm kubectl

These commands will install:

kubelet : kubernetes node runtime
kubeadm : kubernetes cluster setup tool
kubectl : kubernetes cluster command line interface
nfs-common: nfs client library to be able to connect to NFS server

Setup cluster (Master Node only)

This step, we will initialize master node by running command.

# init master
kubeadm init

Note: Run on master node only

You will need to wait for sometimes depending on your internet connection.

If everything is ok, you will see.

Note:
Please copy the last line. It will be used in the next step to join worker node.

Setup cluster (Worker Node only)

Run join command copied from previous step on all the worker nodes.

Here is an example (Do not use this command, it won't work for you)

kubeadm join 192.168.1.109:6443 --token srzyez.wjnqsmt2gcohxtp4 --discovery-token-ca-cert-hash sha256:ac8d21e46aeaba3664c4f6060072de03d906d0032f9731b021eeb8d54a876e35

On worker node, if join command succeed, you will see.

Post install (Master node only)

1. Setup cluster connection configuration

Switch back to normal user.

exit

Run below command and enter root password.

# setup kube config
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

2. Install Pod network (weavenet)

A kubernetes cluster needs pod network. To install weavenet (one of several pod network providers), run command on master node as below.

sudo sysctl net.bridge.bridge-nf-call-iptables=1
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

3. Check the result

On master node, run command below.

kubectl get nodes

If you see something like below, congratulation! you have your cluster.

Install Dashboard (Optional)

To be able to see the overall picture of your cluster, you need a dashboard.

1. Install dashboard

On master node, run below command.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

2. Setup permission to access dashboard

For simplify thing, we will grant admin permission to anyone access dashboard. You won't do this in the actual cluster.

echo "apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system" | kubectl apply -f -

3. Run proxy command

Open a proxy so that the cluster can be accessed from outside.

kubectl proxy --address='0.0.0.0' --accept-hosts='.*'

4. Open dashboard UI

Open url https://<your master node ip>:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy.

You should see login page.

Click "Skip" button and you will see overview page.

Congratulation! You've got a dashboard to see and manage your workload.

Summary

We've got a kubernetes cluster running on VMs. Next step will setup cluster storage so we can build application that store something.

Prepare Virtual Machine using Virtualbox and Ubuntu

pongsatt — Sat, 03 Nov 2018 06:52:42 +0000

This post will guide you to create 4 virtual machine based on Ubuntu Server. One designated to be kubernetes master node and the other three as worker nodes.

This is part of Setup your own kubernetes cluster on VMs series.

Note:

This guide is based on Windows but since it's a virtual machine, others OS shouldn't be different

Prerequisite:

Install VirtualBox
Download Ubuntu 16.04 Image for VirtualBox or 64bit Direct link
Available 2GB of RAM per VM (4 nodes will need at 8GB). You can reduce number of nodes depending on your available RAM
10-30GB of disk space per VM (4 nodes will need around 100GB)

In the end

You will get something like this.

1. Create Virtual Machine

We will create 4 virtual machines. 1 machine is for kubernetes master and the reset for worker node. They're all mostly the same except hostname. RAM and cpu can be adjusted based on your need. In theory, master node will need less resource comparing to worker node.

1.1. Create New Virtual Machine

Name: k8smaster
Type: linux
Memory: 2048MB

1.2. Choose harddisk size

Size: 10GB

2. Prepare VM before install OS

2.1. Setup network

We will use bridge network so we have the real network ip address that can talk to the outside world.

Click "Settings" then "Network".
Select "Attached to" as "Bridged Adapter"
Click "Advance"
Write down "MAC Address" to reserve IP address so that your IP will not change every time the VM restarted

2.2. Setup Ubuntu image to be installed

Click "Settings" then "Storage"
Click "Adds optical drive" at the top right corner
Select os image (.iso) you downloaded
Click OK

3. Create the rest VMs

Repeat step 1 and 2 with below information

Name: kubenode1, kubenode2, kubenode3
RAM: 2048GB (or more)
Disk: 32GB (or less)

4. Setup Ubuntu

4.1 Start VM (kubemaster)

Click "Start" button icon.

Ubuntu setup screen will be shown as below.

Select "Install Ubuntu Server

4.2 Set Hostname

Follow instructions until this screen and fill "kubemaster" as hostname.

4.3 Set user name

Fill in "Full name", "username" and "password" as needed. Please make note these information.

4.4 Other information

"Encrypt your home directory?" : No
"Partition disk" : "Guided - use entire disk"
"Write the changes to disks?" : Yes
"HTTP proxy information" : leave blank (if you are outside company proxy)
"How do you want to manage upgrades on this system?" : No automatic updates
"Software selection" : select "OpenSSH server" (This is required if you want you connect to your server remotely using ssh command)

4.5 Install GRUB boot loader

Select "Yes" then "Finish Installation"

5. Setup Ubuntu for the rest VMs

Repeat step 4 for "kubenode1" to "kubenode3"

Note:

Don't forget to change hostname to "kubenode1" to "kubenode3" in step 4.2

Username and password can be the same for all VMs

Summary

Now, you have all the VMs with network and ready to setup kubernetes cluster in the next step.