DEV Community: Jace

The path to CKA, and some tips.

Jace — Sun, 12 Apr 2020 10:26:34 +0000

Yes, I passed the CKA exam.👍

Since I have already been working with kubernetes for yeas. The way I prepare CKA and the schedule I made might not suit for the beginners. But, still, some of the tips and resources can be shared.

Resources

Certified Kubernetes Administrator (CKA) with Practice Tests

I highly recommend this course. It contains the sufficient lecture for the CKA test. Most of all, the hands-on labs have good coverage for the skill you need for the CKA test.

Tasks tab in official k8s doc. There are lots of hand-on task in the task tab of official document. And some of them is important for the test.

Tips and advice

A least know how to use tmux!
Your test environment might have a chance to be extremely slow.
There was one question that took me a lot of time to wait for the command execution to be finished.
At the first attempt, I wait for about 5 mins. But the command still stuck there. I have no choice but to kill that command and skip to another questions. When I finished the rest of all question, I still have about 15~20 minutes. So I typed the same correct command again and wait for the execution to be finished. But in the end, those 15~20 minutes is not enough.
If I know the basic usage of tmux, I can keep that command being running, in a meanwhile, I can double check my answers.
Get familiar with kubectl run
Get familiar with this command, it can generate yaml templates for pods, depolyments. And save you lots of time.
Tips for The Certified Kubernetes Exams: CKA and CKAD in 2020
Special thanks for Derek who leaved a tips in my first article about preparing CKA.

Cmd and Entrypoint in Docker container.

Jace — Sun, 19 Jan 2020 11:08:25 +0000

For a long time, I can not tell the exact difference between CMD and ENTRYPOINT in Dockerfile.
Until I have a chance to take a look for the definition.

I will explain this base on few different scenarios.

Let's say If I have a Ubuntu image-based container named ubuntu-sleeper to execute the sleep for 5 seconds. And the Dockerfile showed as below.

FROM ubuntu

CMD sleep 5

When I type docker run ubuntu-sleeper, container will automatically execute the sleep 5 to sleep 5 seconds.

But what if I like to customize the second to sleep?

As we know, we can overwrite the default CMD by appending the command after the docker run.
For example, docker run ubuntu-sleeper sleep 10

But it looks bad, and we like make it look like this.
docker run ubuntu-sleeper 10
Which only pass the second argument we need.

To achieve this, we can use ENTRYPOINT, to write our dockerfile..

FROM ubuntu

ENTRYPOINT ["sleep"]

Docker will execute the command in ENTRYPOINT first, then the CMD.
So, when we type docker run ubuntu-sleeper 10.
Docker will get the ENTRYPOINT sleep and the 10 we passed via run commmad.

But what if I run this container without passing the CMD?
For example, just run docker run ubuntu-sleeper.
As we know, this will make container only run sleep without given argument. And you will get the error says the operand is missing.

So how do you define the default value for this container?
In this case, You can define ENTRYPOINT and CMD at a same time.

FROM ubuntu

ENTRYPOINT ["sleep"]
CMD ["5"]

So when you simply run docker run ubuntu-sleeper, the container will sleep 5 seconds by default.
And you can change CMD value by docker run ubutnu-sleeper 20

PS: And you can even change the ENTRYPOINT with --entrypoint option in docker run.

docker run --entrypoint sleep2.0 ubuntu-sleeper 10

TIL: The path to CKA Week 5.

Jace — Sun, 05 Jan 2020 09:13:28 +0000

After finished the Kubernetes installation the hard way,
I moved on the the CKA with Practice Tests that I bought in udemy.

This course contains lots of online hand-on labs, which improve the kubernetes troubleshooting skill.

ETCD

etcd is a key-value storage for all of the data about the whole information of a k8s cluster.

And you can deploy etcd from by downloading a released binary and execute it.
Or, if you install kubernetes cluster by kubeadm, your etcd will run as container. For example, you can get the keys of your cluster by
kubectl exec etcd-master -n kube-system etcdctl get / --perfix -keys-only

kube-apiserver

In kubernetes, there are some operation/component related with kube-apiserver.

Authenticate User
Validate Request
Retrieve data
Interactive with ETCD
Scheduler
Kubelet

You can see your kebe-apiserver configuration by checking cat /etc/systemd/system/kube-apiserver.service or
cat /etc/kubernetes/manifests/kube-apiserver if your cluster is installed by kubeadm
Or you can search a running api-server process by
ps -aux | grep kube-apiserver, and you can see the all configurations.

kube-controller-manager

When install kube-controller-manager there are some different controller will be installed. Eg, development controller, replication controller, etc.

And if your cluster is install by kubeadm, like other components, the kube-controller-manager will run as a pod in kube-system namespace on master node.
For other non-kubeadm setups, the configurations can be checked in /etc/systemd/system/kube-controller-manager.service

kube-scheduler

In k8s, kube-scheduler will decides and only decides which pod goes to which worker node.
By considering the pod's resources requirements, taints, node selectors and affinity, etc.

Again, for a cluster set by kubeadm the kube-scheduler configuration will be in /etc/kubernetes/manifests/kube-scheduler.yaml.
And it run as a pod in the kube-system namespace.

kubelet.

kubelet is where the dirty work happends in kubernetes cluster.
It receives the command from the scheduler and to interact with the container run time to start or to delete a container.
If you use kubeadm to install k8s cluster, it WILL NOT install kubelet.
You must download and install it by your own.

You can search your kubelet process by
ps aux | grep kubelet

kube-proxy.

kube-proxy will setup the iptables for the services in kubernetes.
Since the service ip and the pod ip are in different subnet.
So the kube-proxy's job is to monitor the services and the pod ips, and maintains the proper rules to forward the traffic.

Notes.

Create pod by kubectl run commnad.
kubectl run nginx --image=nginx --restart=Never
Create deployment by kubectl run
kubectl run nginx --image=nginx --restart=Always
Create job by kubectl run
kubectl run nginx --image=nginx --restart=OnFailure

Ansible notes: group_names

Jace — Mon, 30 Dec 2019 06:20:46 +0000

In ansible, group_names is a list (array) of all the groups the current host is in.

This can be used in templates using Jinja2 syntax to make template source files that vary based on the group membership (or role) of the host:

{% if 'webserver' in group_names %}
   # some part of a configuration file that only applies to webservers
{% endif %}

For example, If I must run a task for certain host in the group of hosts.

And the inventory hosts file shown as follows.

[all]
node1  ip=192.168.1.1
node2  ip=192.168.1.2

[delpy_infra]
node1
node2

[special]
node2

And here's the task.
You can use group_names to specify the certain host.

- name: Speciall ops for special nodes
  command: "/usr/bin/special.sh"
  when: "'special' in {{ group_names }}"

TIL: The path to CKA Week 4.

Jace — Tue, 24 Dec 2019 14:49:32 +0000

Last week I finished the course in the CKA bundle.
It's time to move on to the kubernetes installation with the hard way😎

Kubernetes The Hard Way is a tutorial walks you through setting up Kubernetes step by step. Meaning without using kubeadm, kubespray or any other tools.

The prerequisites of this tutorial contains GCP.
Since I'ver never use the GCP before, so I toke this as a opportunity.
After I finished the tutorial, my new-created GCP account had not been charged any cent yet.

I think I'll just skip the detail steps and note the TIL.
For those who interested it this tutorial could just click the link above.
Here's the labs for the tutorial, and also the common steps to setup a kubernetes cluster.

Installing the Client Tools
Provisioning Compute Resources
Provisioning the CA and Generating TLS Certificates
Generating Kubernetes Configuration Files for Authentication
Generating the Data Encryption Config and Key
Bootstrapping the etcd Cluster
Bootstrapping the Kubernetes Control Plane
Bootstrapping the Kubernetes Worker Nodes
Configuring kubectl for Remote Access
Provisioning Pod Network Routes
Deploying the DNS Cluster Add-on

TIL : GCP compute region/zone
At the beginning of the tutorial, I have to prepare the GCP environment.
And there are two commands to set up the region and zone for your GCP compute unit.
A region can be seen as a data center in diffident era of the world.
And the zones are isolated location within a region, kinda like different server in the same data center.
So for the HA deployment, you may have multiple application running in different zone within certain region.

And for those wonder if gcloud command have auto-completion.
You can try

TIL: The path to CKA Week 3.

Jace — Thu, 19 Dec 2019 14:08:28 +0000

TIL: The path to CKA Week 3.

Helm

The helm tool packages a Kubernetes application using a series of YAML files into a chart, or package. This allows for simple sharing between users, tuning using a templating scheme, as well as provenance tracking, among other things.

Helm v2 consists of two components.

A server call tiller, which runs inside the k8s cluster.
A client call Helm, which runs on the local machine or any machine that is able to talk to tiller server.

In helm version 2, uses Tiller pod to call k8s api to deploy the pods. The new Helm v3 does not deploy a tiller pod.

Chart Contents

A chart is an archived set of kubernetes resource manifests that make up a distributed application.

├── Chart.yaml
├── README.md
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── configmap.yaml
│   ├── deployment.yaml
│   ├── pvc.yaml
│   ├── secrets.yaml
│   └── svc.yaml
└── values.yaml

Chart.yaml
The Chart.yaml file contains some metadata about the Chart, like name version, and so on.
values.yaml
The values.yaml contains keys and values that used to generate the release in your cluster. These values are replaced in the resource manifests using the GO templationg syntax.
templates
The templates directory containes the resource manifests that make up the application.

Security

Accessing the API
To perform any action in a Kubernetes cluster, you need to access the API and go through three main steps:

Authentication:
Authorization (ABAC or RBAC):
Admission Control.

Once a request reaches the API server securely, it will first go through any authentication module that has been configured. The request can be rejected if authentication fails or it gets authenticated and passed to the authorization step.

At the authorization step, the request will be checked against existing policies. It will be authorized if the user has the permissions to perform the requested actions. Then, the requests will go through the last step of admission. In general, admission controllers will check the actual content of the objects being created and validate them before admitting the request.

In addition to these steps, the requests reaching the API server over the network are encrypted using TLS. This needs to be properly configured using SSL certificates. If you use kubeadm, this configuration is done for you.

ABAC, BRAC and Webhook

ABAC

ABAC stands for Attribute Based Access Control. It was the first authorization model in Kubernetes that allowed administrators to implement the right policies. Today, RBAC is becoming the default authorization mode.

For example, the policy file shown below authorizes user Bob to read pods in the namespace foobar:

{
    "apiVersion": "abac.authorization.kubernetes.io/v1beta1",
    "kind": "Policy",
    "spec": {
        "user": "bob",
        "namespace": "foobar",
        "resource": "pods",
        "readonly": true     
    }
}

RBAC

RBAC stands for Role Based Access Control.

All resources are modeled API objects in Kubernetes, from Pods to Namespaces. They also belong to API Groups, such as core and apps.These resources allow operations such as Create, Read, Update, and Delete (CRUD), which we have been working with so far. Operations are called verbs inside YAML files. Adding to these basic components, we will add more elements of the API, which can then be managed via RBAC.

Rules are operations which can act upon an API group. Roles are a group of rules which affect, or scope, a single namespace, whereas ClusterRoles have a scope of the entire cluster.

Each operation can act upon one of three subjects, which are User Accounts which don't exist as API objects, Service Accounts, and Groups which are known as clusterrolebinding when using kubectl.

RBAC is then writing rules to allow or deny operations by users, roles or groups upon resources.

While RBAC can be complex, the basic flow is to create a certificate for a user. As a user is not an API object of Kubernetes, we are requiring outside authentication, such as OpenSSL certificates. After generating the certificate against the cluster certificate authority, we can set that credential for the user using a context.

Roles can then be used to configure an association of apiGroups, resources, and the verbs allowed to them. The user can then be bound to a role limiting what and where they can work in the cluster.

Here is a summary of the RBAC process:

Determine or create namespace
Create certificate credentials for user
Set the credentials for the user to the namespace using a context
Create a role for the expected task set
Bind the user to the role
Verify the user has limited access

Finally, the last quiz at the end of this course.😂

TIL: The path to CKA Week 2.

Jace — Mon, 16 Dec 2019 12:19:45 +0000

Same as last week, keep working on the tutorial of Kubernetes Fundamentals.

Services

Service in kubernetes is a very important object. They are agents which connect Pods together, or provide access outside of the cluster.
The common types of service are ClusterIp, NodePort and LoadBalancer.

Here's the official explanation.

ClusterIp

The ClusterIP service type is the default, and only provides access internally (except if manually creating an external endpoint). The range of ClusterIP used is defined via an API server startup option.

NodePort

The NodePort type is great for debugging, or when a static IP address is necessary, such as opening a particular address through a firewall. The NodePort range is defined in the cluster configuration.

LoadBalancer

The LoadBalancer service was created to pass requests to a cloud provider like GKE or AWS. Private cloud solutions also may implement this service type if there is a cloud provider plugin, such as with CloudStack and OpenStack. Even without a cloud provider, the address is made available to public traffic, and packets are spread among the Pods in the deployment automatically.

And the last one ExternalName, which is new to me. 💡

A newer service is ExternalName, which is a bit different. It has no selectors, nor does it define ports or endpoints. It allows the return of an alias to an external service. The redirection happens at the DNS level, not via a proxy or forward. This object can be useful for services not yet brought into the Kubernetes cluster. A simple change of the type in the future would redirect traffic to the internal objects.

In short, externalName can point to external DNS CNAME, not like the other service will point to a certain pod.

Volumes and Data

There's one project call Rock mentioned in this section.
Rock can be seem as a storage orchestrator provides a common framework across all of the storage solutions, like ceph, nfs, Cassandra, etc.

Rook turns distributed storage systems into self-managing, self-scaling, self-healing storage services. It automates the tasks of a storage administrator

Scheduling

Scheduling is the key of kubernetes.
There's a lot of strategy that you can use when scheduling pods in kubernetes.
One is call pod affinity, and there are several kind of pod affinity rules.

requiredDuringSchedulingIgnoredDuringExecution
means that the Pod will not be scheduled on a node unless the following operator is true. If the operator changes to become false in the future, the Pod will continue to run. This could be seen as a hard rule.
preferredDuringSchedulingIgnoredDuringExecution

preferredDuringSchedulingIgnoredDuringExecution will choose a node with the desired setting before those without. If no properly-labeled nodes are available, the Pod will execute anyway.

podAffinity / podAntiAffinity
Which will keep pods together or keep them on different nodes.
topologyKey
This rule is new to me.
The topologyKey allows a general grouping of Pod deployments. Affinity (or the inverse anti-affinity) will try to run on nodes with the declared topology key and running Pods with a particular label. The topologyKey could be any legal key, with some important considerations.
In short, we can use topologyKey to scheduler our pod based on the topoloy in a rack or a region's view.

💡To check the scheduler event on the current k8s cluster, you can use.

kubectl get events

Logging and Troubleshooting

Ephemeral Containers
Ephemeral container is a new alpha feature in v1.16.
Allowing user can attach a container to a existing pod.
You may be able to use the kubectl attach command to join an existing process within the container. This can be helpful instead of kubectl exec, which executes a new process. The functionality of the attached process depends entirely on what you are attaching to.

kubectl debug buggypod --image debian --attach

Cluster Start Sequence
If you built the cluster using kubeadm, then the sequence will begins with systemd.
kubelet will create any YAML inside the staticPodPath.
So this is also where that api-server, kube-scheduler, kube-controller YAMLs stored.
In most case your staticPodPath will be /etc/kubernetes/manifests
So you can see your api-server, kube-scheduler or other default service's YAMLs here.

TIL: The path to CKA Week 1.

Jace — Mon, 09 Dec 2019 13:33:42 +0000

This week, I started the course of k8s Fundamentals that I bought with CKA bundle.

The course starts from the history and the basic of container orchestrator, which I'm quite familiar with. So I didn't spend too much time on it.

And follows by the Kubernetes Architecture section. Telling what components that run on k8s cluster and how they work.
For me, this is just a quick review, nothing new.
So I just went through this section and took 100% on knowledge check of this section.

To do the lab in the course I have to prepare a k8s environment, so I started 2 VMs by VirtualBox on my PC.
There is a Lab during the course that teach you how to install a k8s by kubeadm.
If you plan to setup a experience environment by VirtualBox, remember to turn off the swap.
Just a little complain, the document of this CNCF still have room for improvement. Noting bash and YAML codes by pdf format is not a decent way.

After I finished k8s installation, I moved on to the API and Access section.
And there is a TIL here 😀.
You can use --v to set the verbose level for kubectl

kubectl --v=6 get nodes

You can even see how kubectl send the curl to the api-server.

Also, I notice that there is a namespace called kube-node-lease created by default.
And kube-node-lease is to keep the worker node's lease information.
Here's the introduction of lease from official document

In versions of Kubernetes prior to 1.13, NodeStatus is the heartbeat from the node. Node lease feature is enabled by default since 1.14 as a beta feature (feature gate NodeLease, KEP-0009). When node lease feature is enabled, each node has an associated Lease object in kube-node-lease namespace that is renewed by the node periodically, and both NodeStatus and node lease are treated as heartbeats from the node. Node leases are renewed frequently while NodeStatus is reported from node to master only when there is some change or enough time has passed (default is 1 minute, which is longer than the default timeout of 40 seconds for unreachable nodes). Since node lease is much more lightweight than NodeStatus, this feature makes node heartbeat significantly cheaper from both scalability and performance perspectives.

API Objects

You can turn on and off the scheduling to a node with the kubectl cordon/uncordon commands.

kubectl cordon -h
Mark node as unschedulable.

Examples:
  # Mark node "foo" as unschedulable.
  kubectl cordon foo

Options:
      --dry-run=false: If true, only print the object that would be sent, without sending it.
  -l, --selector='': Selector (label query) to filter on

Usage:
  kubectl cordon NODE [options]

I've been using taint to turn on/off scheduling a node for a long time.
This is quite handy. Good to know this.

So far, I've finished 7 section out of 17 from CNCF k8s Fundamentals.

I must say CNCF training system have lots of room for improvement.
Some of the quiz just broken, some of them with incorrect answer which will misleading the new learner.
Some of them even have no answer, the system just kept telling me I was wrong even I tried the whole options. I've reported 2 issues in the class forum.

TIL: The path to CKA Week 0.

Jace — Sat, 07 Dec 2019 06:46:54 +0000

I've been using Kubernetes and working with kubernetes related project for about 2 yesrs.
So, it's time to set a goal to test myself for the understanding of k8s; and CKA is a very iconic certification for the kubernetes admins.

So I setup a goal for myself to pass CKA as soon as possible, since I would like to take other certification exam in the future.

Let's start with my background.
I've been using container related technique for about 3~4 years.
And my current work is about build up NFVI based on the kubernetes platform.
In short. I'm familiar with kubernetes deployment/management and able to using several kinds of plugin like multus, sriov-cni, cmk, nfd, etc.

But, it been about 2 years since the last time I built up k8s all by manually with k8s v1.9. So it's better for me to prepare CKA from the start.

During the Black Friday and the Cyber Monday.
I bought the CKA + k8s Fundamentals and Certified Kubernetes Administrator (CKA) with Practice Tests on udemy.

And I plan to start from k8s Fundamentals course, then the Certified Kubernetes Administrator (CKA) with Practice Tests.
Also, I will go through the k8s hard way.
https://github.com/kelseyhightower/kubernetes-the-hard-way
Finally, exercise the tasks part in k8s.io document.
Last but not least, scheduling for CKA exam.

So, let's go. Wish me luck. 🤘

Horizontal Pod Autoscale with Custom Prometheus Metrics🚀

Jace — Fri, 01 Nov 2019 09:46:50 +0000

0. 前言

這篇文章會分享如何在Kubernete上透過自訂的Prometheus metrics來實作HPA
-- Horizontal Pod Autoscale

首先來講講為什麼會需要custom metrics
很簡單，就是內建的不夠用
其實單純用內建的CPU/Ram作為Scale的指標其實是非常不準確的一件事情。

可參考
Cpu utilization is wrong
Useless metrics

所以針對一些原生K8s沒有提供的指標，像是使用服務的人數，連線的latency等等。
就必須使用Custom Metrics來執行Scaling.

下列操作基於於kubernetes v1.14

1. HPA via Custom Metrics

免不了的，一定得解釋一下HPA透過Custom Metrics做Scaling的機制
單純透過CPU or Ram其實沒甚麼好講的，官方文件跟網路上很多文章講解的都很清楚
~~其實是我懶得寫~~
k8s.doc
ithome

重點在於HPA是如何取得Custom Metrics的，以及如何將我們自訂的Metric吐出來給HPA Controller

大致運作邏輯是這樣

首先講解Custom Metrics的部分，在Kubernetes中，目前有關Custom Metrics的操作主要是對

custom-metrics.metrics.k8s.io/v1beta1

這個API EndPoint來註冊以及取得資料的。

也就是說，我們的Pod暴露出的Custom Metrics要想辦法註冊到這個API Endpoint上
HPA Controller就可以從這個EndPoint取得我們自定的Metrics，來做他該做的事情

2. Prometheus Operator & Prometheus-adapter

先從架設環境開始，在這邊我會透過Prometheus來集中接收Metrics
應用服務對Prometheus暴露Metrics
而會有個Prometheus-adapter把metrics註冊到上述的Endpoint

其實也可以讓應用直接對K8s的Custom API Server直接註冊Custom Metrics
但統一將Metrics 交給Prometheus後，可以更方便我們整合Grafana Dashboard

2.1 安裝Prometheus Operator

Prometheus Operator是由coreOS開發打包推出的
以下是官方的簡介

The Prometheus Operator for Kubernetes provides easy monitoring definitions for Kubernetes services and deployment and management of Prometheus instances.

和一般Prometheus最大的不同，Prometheus Operator會順帶安裝Grafana
並且還多加入了一個Service Monitor

Service Monitor會負責將Pod暴露出來的metric導入到Prometheus內
有點類似prometheus的scrape

我們可以直接用helm安裝prometheus operator

helm install --name po stable/prometheus-operator

安裝好後可以連進prometheus的網頁查看所有的Target是否都能正確取得
如果Kube-proxy有異常的話可參考這邊

2.2 安裝Prometheus-adapter

安裝完Pormetheus Operator後我們打通了Pod export metrics到Prometheus的管道
但還必須將Prometheus內的metrics註冊到k8s的custom metrics API endpoint

同樣的可以透過helm來安裝prometheus-adatper

helm install --name pa stable/prometheus-adapter

要額外注意的一點是prometheus.url和prometheus.port必須配合我們剛剛安裝好的prometheus operator.
可以在安裝的時候指定這些參數

helm install --name pa --set prometheus.url=http://po-prometheus-operator-prometheus.default.svc,prometheus.port=9090 stable/prometheus-adapter

安裝好後可以透過這個指令來從custom metrics api endpoint撈資訊
有噴出一堆東西的話就是安裝成功了

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

如果有回傳，但沒看到metrics。表示你的Prometheus Operator有安裝成功，
正確註冊了Custom metrics api endpoint，但Adapter沒有從Prometheus撈到資料註冊回去。

至此為止，加上Prometheus Operator以及Prometheus adapter後
原先的的關係圖會變成這樣子

接下來會介紹如何將metrics expose到Prometheus內

3. Pod exposes custom metrics.

Pod暴露metrics的方法大致可分為幾種

透過lib自行輸出metrics 例如 flask_prometheus_metrics 就是會幫你輸出flask裡面的相關指標
自己按照Prometheus所需的格式輸出metrics 例如我們接下來用的範例就是如此
由Ingress或Service Mesh幫你暴露metrics (可自訂性較低)

3.1 Start a sample Pod.

Create sample-deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - image: luxas/autoscale-demo:v0.1.2
        name: metrics-provider
        ports:
        - name: http
          containerPort: 8080

kubectl apply -f sample-deploy.yaml

Create svc.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: sample-app
    release: po
  name: sample-svc
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: sample-app
  type: ClusterIP

kubectl apply -f svc.yaml

等pod建立好後，可以透過下列指令來觀察pod的metrics

curl http://$(kubectl get service sample-svc -o jsonpath='{ .spec.clusterIP }')/metrics

Output >>>
# HELP http_requests_total The amount of requests served by the server in total
# TYPE http_requests_total counter
http_requests_total 1

3.2 ServiceMonitor

如先前介紹的，Service Monitor主要負責將Pod吐出的資料倒到Prometheus內
類似Scrape的方式

Create service-monitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: sample-app
    release: po
  name: sample-app
  namespace: default
spec:
  endpoints:
  - port: http
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app: sample-app
      release: po

kubectl create -f service-monitor.yaml

到這一步需要開啟Prometheus的Dashboard檢查Target內有沒有我們定義的Target
這裡附上ServiceMonitor的除錯方法
其實不外乎就是label沒對上罷了

4. Double Check

做到這邊，我們整個路算是都打通了。
可以從apiserver撈一下，檢查我們的metric有沒有註冊上去

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests?selector=app%3Dsample-app"

註: 我們安裝prometheus adapter時只用了預設的config，如果你的metric沒有出現。
或是希望在註冊回k8s時要做一些運算，如算rate之類的。
可以參考這邊

5. Create HPA

搞了這麼久終於可以來定義我們的HPA了。

Create and apply sample-hpa.yaml

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: sample-app
spec:
  scaleTargetRef:
    # point the HPA at the sample application
    # you created above
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  # autoscale between 1 and 10 replicas
  minReplicas: 1
  maxReplicas: 10
  metrics:
  # use a "Pods" metric, which takes the average of the
  # given metric across all pods controlled by the autoscaling target
  - type: Pods
    pods:
      # use the metric that you used above: pods/http_requests
      metricName: http_requests
      # target 500 milli-requests per second,
      # which is 1 request every two seconds
      targetAverageValue: 500m

And try to get our hpa

kubectl get hpa
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
sample-app   Deployment/sample-app   0/500m    1         10        1          24h

6. Trigger Scale Up.

先開兩個視窗分別觀察HPA以及deployment

watch kubectl get hpa

watch kubectl get deploy sample-app

接著再打一個curl的無窮迴圈

while true; do  curl -s "http://$(kubectl get service sample-svc -o jsonpath='{ .spec.clusterIP }')" > /dev/null; done

沒意外的話就會看到HPA被觸發然後進行Scale Up了

7. Misc.

最先是打算做http request latency based HPA的，所以
拿了flask_prometheus_metrics來做範例
其實最後也算是有弄出來，但中間會牽扯到Prometheus adapter的metricsQuery語法撰寫
礙於篇幅的關係就不寫了。
而且發現Prometheus adapter的metricsQuery其實還是有侷限性。
他只能針對一個metric來做rate或是histogram，然後再註冊到K8s custom api endpoint
不像我們用grafana時，可以拿很多metric組合運算得出我們想要的數據

所以就直接用這個例子分享。

結論，最好的方法還是Pod自己就能吐出最終你要註冊到k8s的metric

TroubleShoot

Prometheus Operator重裝失敗
再重裝Prometheus Operator時可能會遇到CRD相關的錯誤
先用Helm delete --purge的方式清空
再刪除所有的crd
https://github.com/coreos/prometheus-operator#removal
然後重新安裝時先手動建立CRD

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/podmonitor.crd.yaml

接著helm install時不要產生CRD

helm install --name po stable/prometheus-operator --set prometheusOperator.createCustomResource=false

我的Prometheus Dashboard內kube-proxy target錯誤

 https://github.com/helm/charts/tree/master/stable/prometheus-operator#kubeproxy

DEV Community: Jace

The path to CKA, and some tips.

Resources

Tips and advice

Cmd and Entrypoint in Docker container.

TIL: The path to CKA Week 5.

ETCD

kube-apiserver

kube-controller-manager

kube-scheduler

kubelet.

kube-proxy.

Notes.

Ansible notes: group_names

TIL: The path to CKA Week 4.

TIL: The path to CKA Week 3.

Helm

Chart Contents

Security

ABAC, BRAC and Webhook

ABAC

RBAC

TIL: The path to CKA Week 2.

Services

Volumes and Data

Scheduling

Logging and Troubleshooting

TIL: The path to CKA Week 1.

API Objects

TIL: The path to CKA Week 0.

Horizontal Pod Autoscale with Custom Prometheus Metrics🚀

0. 前言

1. HPA via Custom Metrics

2. Prometheus Operator & Prometheus-adapter

2.1 安裝Prometheus Operator

2.2 安裝Prometheus-adapter

3. Pod exposes custom metrics.

3.1 Start a sample Pod.

3.2 ServiceMonitor

4. Double Check

5. Create HPA

6. Trigger Scale Up.

7. Misc.

TroubleShoot

Reference