DEV Community: Cosmas Nyairo

Gateway VPC Endpoints on AWS

Cosmas Nyairo — Sun, 05 May 2024 10:10:33 +0000

For our services, if we want to have internet access, we route the traffic via an internet gateway.

However, there may be cases where we we don't want our network traffic to go through the public internet, we could utilise vpc endpoints, with vpc endpoints, we are able to access aws services via a private network.

A gateway vpc endpoint targets ip routes in a prefix list that belong to an aws service. Supported services are: AWS S3 and DynamoDB

Below architecture diagram of a gateway vpc endpoint showcases the implementation:

Gateway VPC Endpoints Example:

Guide to creating a gateway vpc endpoint.

On the vpc dashboard click on the endpoints tab, then create endpoint button on the right.

Choose the name of the endpoint being created, For this demo, we use the aws services category,

Use the filter: Type = Gateway for us to create a gateway vpc endpoint then, choose the service name type to be created then choose your private vpc and which route table for the prefix list entries to be added to.

Choose the VPC endpoint policy to be used when accessing the resources and tag your resources for effective cost tracking.

Our route table will have new entries added with the prefix list of the service we're connecting to:

Certified Kubernetes Administrator (CKA) Prep.

Cosmas Nyairo — Wed, 06 Dec 2023 19:41:17 +0000

Hey there,

For my CKA preparation, I have noted down topics i studied for in preparation for the exam as well as commands I use on a day to day basis.

N/B:

Most of the commands here I run them on a cloud9 instance whitelisted to access to my test EKS cluster.

A master/worker node setup on ec2 would work as well. Will share a guide on how that can be achieved.

Notes
Scheduling
- Manual Scheduling
- Labels
- Taints and Tolerations
- Node Selector
- Node affinity
- Resource Limits
- Daemon Sets
- Static Pods
- Multiple Schedulers
- Scheduler Profiles
Logging and Monitoring
OS Upgrades
Security
- Authentication
- TLS Certificates
- Kubeconfig
- Authorization
- Network Policy
Storage
Networking -Network configuration - Cluster Nodes -Pod Networking -Service Networking -DNS
JSONPATH

Notes

Useful commands and aliassed for the terminal:

# in .bashrc
alias k="kubectl"
export drc="--dry-run=client -oyaml"
export drs="--dry-run=server -oyaml"
export kre="kubectl replace --force -f"
# in .vimrc
set nu
set autoindent

kubectl explain po.spec

etcd - key store for information about the cluster i.e nodes, pods, roles, secrets
kube-api sever - management component in kubernetes. Only component to interact with the etcd data store
kube-controller-manager - manages contollers in kubernetes and brings the system to the desired state
kube-scheduler - decides which pod goes to which node
kubelet - registers nodes, creates pods on a nodes, monitors the pods and nodes
kube-proxy - creates forwarding rules for nodes in the cluster

If we install the cluster using kubeadmin, we can view :
  - Exec into the etcd pod and run the `etcdctl` command
  - Exec into the kube-apiserver pod and access the `/etc/kubernetes/manifests/kube-apiserver.yaml` file
  - Exec into the kube-controller manager pod and access the `/etc/kubernetes/manifests/kube-controller-manager.yaml` file
  - Exec into the kube-scheduler pod and access the `/etc/kubernetes/manifests/kube-scheduler.yaml` file
  - Kubeadmin doesn't deploy the kubelet, we have to do it manually

If we install the clustrer without using kubeadmin, we can view:
  - explore the etcd service
  - the apiserver service located at `/etc/systemd/system/kube-apiserver.service` or `ps -aux | grep kube-apiserver` on the master node.
  - the kubecontroller manager service located at `/etc/systemd/system/kube-controller-manager.service` or `ps -aux | grep kube-controller-manager` on the master node.
  - explore the kube-scheduler service or `ps -aux | grep kube-scheduler`
  - explore the kubelet service or `ps -aux | grep kubelet`

Kubelet config is at: /var/lib/kubelet/config.yaml for each node
Test commands

kubectl run webapp-green --image=test --dry-run=server -oyaml

kubectl auth can-i

To pass commands to pods on creation:

kubectl run webapp-green --image=test --command -- --color=green

To pass args to pods on creation:

 kubectl run webapp-green --image=test -- --color=green

Update env variables:

kubectl -n test set env deploy/test --containers=container-name DB_Host=mysql DB_User=user1 DB_Password=pass123

Check certificate:

openssl x509 -in /opt/cert.crt -text

Check service logs:

journalctl -u service

service servicename status

Kubelet file located at: var/lib/kubelet/config.yaml

Scheduling

Manual scheduling

Add the nodeName field when creating a pod only during creation time
We can alternatively create a pod binding file as shown below:

apiVersion: v1
kind: Binding
metadata:
  name: myapp
target:
  apiVersion: v1
  kind: Node 
  name: my-node

Then, we send a post request to the pod's binding api with data section set to the binding object in a json format as follows:

curl --header "Content-Type:application/json" --request POST --data <<BINDING-OBJECT-YAML>> http://$SERVER/api/v1/namespaces/<namespace>/pods/$PODNAME/binding/

Labels and Selectors

Annotations record other details in pod for informatory purposes

  kubectl get po --selector type=test

Taints and Tolerations

Restrict what pods can be scheduled on a node
Taints set on nodes, tolerations set on pods
Taint effect = what happens to pods that can't tolerate the taint
- NoSchedule
- PreferNoSchedule
- NoExecute

Taint and toleration example yaml shown below:

apiVersion: v1
kind: Pod
metadata:
  name: <<pod-name>>
spec:
  containers:
    - name: nginx-container
      image: nginx
  tolerations:
    - key: "type"
      value: "test"
      operator: "Equal"
      effect: "NoSchedule"

  kubectl taint nodes my-node type=test:NoSchedule

  kubectl taint node my-node node-role=kubernetes.io/control-plane:NoSchedule-

Node Selector

Add node selector section in the pod with the label of the node we want to use as shown below:

apiVersion: v1
kind: Pod
metadata:
  labels:
    name: myapp
spec:
  containers:
  - name: myapp
    image: busybox
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"
    ports:
      - containerPort: 80
  nodeSelector:
    type: large

  kubectl label nodes my-node type=test

Node affinity

Node affinity example below:

apiVersion: v1
kind: Pod
metadata:
  name: <<pod-name>>
  labels:
    app: <<label-value>>
    type: <<label-value>>
spec:
  containers:
    - name: nginx-container
      image: nginx
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: type
                operator: Exists
          # - matchExpressions:
          #     - key: type
          #       operator: In
          #       values:
          #         - test

Node affinity types:

requiredDuringSchedulingIgnoredDuringExecution
PreferredDuringSchedulingIgnoredDuringExecution
requiredDuringSchedulingrequiredDuringExecution

	During Scheduling	During Execution
1	Required	Ignored
2	Preferred	Ignored
3	Required	Required

We can combine node affinity with taints and tolerations

Resource Limits

We can create limit ranges which need to be followed by pods in a namespace as shown below:

apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-limit-range
spec:
  limits:
    - default:
        cpu: 1
      defaultRequest:
        cpu: 0.5
      type: Container
---
apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range
spec:
  limits:
    - default:
        memory: 512Mi
      defaultRequest:
        memory: 256Mi
      type: Container
---

Daemon Sets

Ensures a copy of a pod is always on each node in the cluster. Use case: Logging, monitoring agents as shown below:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: mydaemonset
spec:
  selector:
    matchLabels:
      app: mydaemonset-app
  template:
    metadata:
      labels:
        app: mydaemonset-app
    spec:
      containers:
      - name: myapp
        image: logviewer

Static Pods

Pods created independently by the kubelet without help from the kubeapiserver.
We can do this in the following ways:

find the kubelet service config path:

ps -aux | grep -i kubelet | grep -i config

check the node's /var/lib/kubelet/config.yaml path
Configure the pod definition path in the kubelet service at with the following entry: --pod-manifest-path=etc/kubernetes/manifests
Provide a --config=mydefinition.yaml entry in the kubelet service and have the mydefinition.yaml file contain the following:
```
  staticPodPath: etc/kubernetes/manifests
```
For both approaches we then place the pod definition files at the specified directory in our case it's /etc/kubernetes/manifests and view pods created via the docker ps command

We can use static pods to deploy control plane components as static pods.

Multiple Schedulers

We can create our custom scheduler in kubernetes then pass the file to the custom scheduler in this format:
--config=/etc/kubenetes/my-scheduler/my-scheduler-conf.yaml.

The my-scheduler-conf.yaml file contains the following:

apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: my-scheduler
leaderElection:
  leaderElect: true
  resourceNamespace: kube-system
  resourceName: lock-object-my-scheduler

The leaderelection section is used when we have multiple master nodes with the custom scheduler running on them in a HA(High Availability) setup and we need to chose the active copy of the scheduler to be running at a time.

Scheduler Profiles

We can disable existing scheduler plugins as well as enable our own custom plugins as below

apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: my-scheduler
    plugins:
      filter:
        disabled:
          - name: TaintToleration
        enabled:
          - name: MyCustomPlugin
  - schedulerName: my-scheduler-two
    plugins:
      score:
        disabled:
          - name: 'NodeAffinity'
          - name: 'ImageLocality'

Monitoring and Logging

We can use the metric server from: metrics-server to collect metrics about our cluster performance. We can also used advanced tools for out monitoring i.e prometheus using this guide

We can then view performance using

kubectl top node

kubectl top pods

To get streamed logs of a currently running pod we use

kubectl logs pod-name -f

To get streamed logs of a currently running pod with multiple containers we use

kubectl logs pod-name -c container-name -f

Os Upgrades

We can drain a node to remove the pods currently running on it and then the node is cordoned (marked as unschedulable)

kubectl drain node1

We can also manually cordon a node

kubectl cordon node1

After our upgrade/fixes to the node, we then uncordon the node once it's back up

kubectl uncordon node1

We can have the different components with different versions but not a higher version than the kube api-server but kubectl can be one version higher or lower or the same version

Component	Maximum version
Kube Api Server	x
Controller Manager	x-1
Kube Scheduler	x-1
Kubectl	x-1 or x or x+1
Kubelet	x-2
Kube-Proxy	x-2

To upgrade a cluster managed by kubeadm,
- we first start with the master node then the worker nodes.
- We first get the upgrade plan for our version,

Step 1 (master node):

kubeadm upgrade plan

apt-get upgrade kubeadm=<version> -y

kubeadm version

kubeadm upgrade apply <version>

If we have kubelets on the master node:

apt-get upgrade kubelet=<version> -y

systemctl restart kubelet

Step 2 (worker node-> one by one):

kubectl drain node01 # on master node

apt-get upgrade kubeadm=<version> -y

apt-get upgrade kubelet=<version> -y

systemctl restart kubelet

kubectl uncordon node01  # on master node

Backup candidates : Resource configurations, etcd

For resource configurations:

  kubectl get all --all-namespaces -o yaml > deploy-and-service.yaml

For Etcd, We can either backup the directory where etcd stores to or take a snapshot and restore it

  export ETCDCTL_API=3

  etcdctl snapshot save --endpoints=<enpoints> \
                        --cacert=<path-to-cacert> \
                        --cert=<path-to-cert> \
                        --key=<path-to-key>
                        /path/to/backup/file/mysnapshot.db

  etcdctl snapshot status /path/to/backup/file/mysnapshot.db

  service kube-api-server stop

  etcdctl snapshot restore /path/to/backup/file/mysnapshot.db --data-dir /path/to/restore/to/
  # We then update etcd configuration to use the new directory

  service etcd restart
  service kube-api-server start

We can use scp to copy objects to a pod:

scp test.txt my-server:/home

scp my-server:/home test.txt

To join cluster:

kubeadm token create --print-join-command

More info: Disaster Recovery for Kubernetes Clusters

Security

Authentication

Authentication to the kube-api server is through:

Password file:
- Specify in kubeapiserver service or kubeapi server yaml file --basic-auth-file=usercreds-basic.csv
- The csv file should be in the fomat of:
```
password,username,userid,groupid
```
- If using api to authenticate pass via curl -v -k https://masternod-ip:port/api/vi/pods -u "user1:password1"
Token file:
- Specify in kubeapiserver service or kubeapi server yaml file --token-auth-file=usercreds-token.csv
- The csv file should be in the fomat of:
```
token,username,userid,groupid
```
- If using api to authenticate pass as a bearer token curl -v -k https://masternod-ip:port/api/vi/pods --header "Authorization: Bearer <TOKEN>"
Certificates
External Identity Service

TLS Certificates

Cerificates with public key usually end with .crt or .pem while those with private keys usually end with .key or -key.pem

List of certificates in the cluster

Certificate Authority certs (we can use more than one ca)
Server certificates:
- kube-api-server
- etcd server
- kubelet server
Client certificates:
- users (admins)
- kube-scheduler
- kube-controller-manager
- kube-proxy
- kube-api-server to kubelet
- kube-api-server to etcd

All certificate operations done by the kube controller manager. We specify the following:

    - --cluster-signing-cert-file='pathtofile'
    - --cluster-signing-key-file='pathtofile'

Certificate signing requests using the following file:

cat file.csr | base64 -w 0

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: johndoe
spec:
  request: BASE64 ENCODED CSR
  signerName: kubernetes.io/kube-apiserver-client
  groups: 
    - system:authenticated
  expirationSeconds: 86400  # one day
  usages:
  - server auth
  - key encryptment

kubectl get csr

kubectl certificate approve csr-name

kubectl certificate deny csr-name

kubectl get csr csr-name -o yaml

Kubeconfig

We can send request using kubectl command without kubeconfig as follows:

kubectl get pods --server server --client-key client.key --client-certificate client.crt --certificate-authority ca.crt

We can use a kubeconfig file to remove the need to add the details and send a request like:

kubectl get pods

Kubeconfig file has these sections:

Clusters (dev,prod)
Users (developer, admin, qa)
Contexts (user-to-cluster i.e developer@dev, admin@prod)

Sample kubeconfig file:

apiVersion: v1
kind: Config
current-context: developer@localcluster-context 
clusters:
- name: remotecluster
  cluster:
    certificate-authority: path/remote-ca.crt
    server: https://ip:port
- name: localcluster
  cluster:
    certificate-authority-data: BASE64 ENCODED CSR
    server: https://ip:port
contexts:
- name: kube-admin@localcluster-context 
  context:
    cluster: remotecluster
    user: kube-admin
    namespace: kube-system
- name: developer@localcluster-context 
  context:
    cluster: localcluster
    user: developer
users:
- name: kube-admin
  user:
    client-certificate: path/admin-client.crt
    client-key: path/client.key
- name: developer
  user:
    client-certificate: path/dev-client.crt
    client-key: path/client.key

kubectl config view 
kubectl config view --kubeconfig path-to-config

kubectl cluster-info --kubeconfig path-to-config

kubectl config use-context developer@localcluster-context

kubectl config set-credentials user --client-certificate=/path/user.crt

Authorization

We set authorization mode in the kubeapiserver entry --authorization-mode=Node,RBAC and they are used following the order they were specified

Role file:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "watch", "list"]
    resourceNames:
      - "devhour"
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch"]

Role binding file:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer-binding
subjects:
- kind: Group
  name: developers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role #this must be Role or ClusterRole
  name: developer # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

kubectl get roles 
kubectl get rolebindings

kubectl auth can-i get pods

kubectl auth can-i get pods --as developer --namespace dev

kubectl create role rolename --verb=create,list,delete --resource=pod

kubectl create rolebinding rolebindingname --role=rolename --user=user-to-bind

To view namespaced resources: --namespaced=true
We can use cluster roles on namespaced resources to give access to the objects in the whole cluster

Cluster Role file:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-admin
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "watch", "list"]

Cluster Role binding file:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-binding
subjects:
- kind: Group
  name: admin
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole #this must be Role or ClusterRole
  name: cluster-admin # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

kubectl create clusterrole clusterrolerolename --verb=* --resource=pod

kubectl create clusterrolebinding clusterrolebindingname --clusterrole=clusterrolerolename --user=user-to-bind

For service accounts:

kubectl create sa service-account-name`

kubectl create token service-account-name

For registry secrets:

 kubectl create secret docker-registry NAME --docker-username=user --docker-password=password --docker-email=email --docker-server=string

Network Policy

By default kubernetes allows traffic from all pods to all destinations.
We can define egress and ingress rules to block traffic to and from pods.

If we add ingress rule, we don't need to specify the egress rule for the response as it is allowed automatically.
We do need an egress rule if we need to send external api calls

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: network-policy-name
spec:
  podSelector:
    matchLabels:
      test: testing # pods matching this label will have the network policy applied on them
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        # The pod selector and namespace selector rules both need to be met.
        # We can add a - before the namespace selector to make it that either of them need to be met.
        - podSelector:
            matchLabels:
              name: pod-name # allow pods with this label to send ingress traffic
          namespaceSelector:
            matchLabels:
              name: testing # specify pods from which namespace are allowed to to send ingress traffic
        - ipBlock:
            cidr: ip-cidr  # range of ip addresses allowed to send ingress traffic
        ports:
          - protocol: "TCP"
            port: 3306 # port to recieve ingress traffic
  egress:
      - to:
          - podSelector:
              matchLabels:
                name: pod-name  # send egress traffic to pods with this label 
            namespaceSelector:
              matchLabels:
                name: testing # send egress traffic to pods in the namespace with this label
          - ipBlock:
            cidr: ip-cidr  # send egress traffic to this range of ip addresses
          ports:
            - protocol: "TCP"
              port: 80 # port to send egress traffic to

Storage

To create a volume in a pod:

apiVersion: v1
kind: Pod
metadata:
  name: <<pod-name>>
  labels:
    app: <<label-value>>
    type: <<label-value>>
spec:
  containers:
    - name: nginx-container
      image: nginx
      volumeMounts:
        - mountPath: /opt
          name: test-volume
        - mountPath: /home/
          name: ebs-volume
  volumes:
    - name: test-volume
      hostPath:
        path: /home/data
        type: Directory
    - name: ebs-volume
      awsElasticBlockStore:
        volumeID: <<vol-id>>
        fsType: "ntfs"

Persistent Volumes: A cluster wide pool of storage volumes which we can assign to applications on the cluster.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: persistent-vol
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  awsElasticBlockStore:
    volumeID: <<vol-id>>
    fsType: "ntfs"

Persistent Volumes Claim: Select storage from the persistent volumes created.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: persitent-vol-claim
spec:
  resources:
    requests:
      storage: 500Mi
  accessModes:
    - ReadWriteOnce

Using Persistent Volumes Claim in a pod:

apiVersion: v1
kind: Pod
metadata:
  name: <<pod-name>>
  labels:
    app: <<label-value>>
    type: <<label-value>>
spec:
  containers:
    - name: nginx-container
      image: nginx
      volumeMounts:
        - mountPath: /opt
          name: pvc-volume
  volumes:
    - name: pvc-volume
      persistentVolumeClaim:
        claimName: persitent-vol-claim

Storage Classes: We don't need to creat a persitent volume. This is done automatically

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-storage
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  fsType: ext4
  encrypted: true

Persitent volume storage class:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: persitent-vol-claim
spec:
  resources:
    requests:
      storage: 500Mi
  storageClassName: ebs-storage
  accessModes:
    - ReadWriteOnce

Networking

Network configuration - Cluster Nodes

Ports required for kubernetes: Ports

Find network interfaces:

ip a

Find bridges in network:

ip a show type bridge

List routes in the host

ip route

Get active internet connections

netstat -nplt

Get number of client connections on a program running on a port

netstat -anp | grep 'etcd\|2379' | wc -l

Pod Networking

When a container is created, the kubelet looks at the cni configuration passed:

network-plugin=cni
cni-conf-dir=/etc/cni/net.d to find our scipts name
cni-bin-dir=/opt/cni/bin to find the script

The kubelet then runs the script with the add command with the name and namespce id of the container net-script.sh add <container> <namespace>

Service Networking

Cluster Ip services are not bound to a specific node but are available to all pods cluster wide
Node Port services exposes application on a port on all nodes in the cluster.

Kube proxy creates forward rules in all nodes such that when we try to reach the ip of a service, the request is forwarded to the ip and port of the pod.

For kubeproxy, we set the proxymode : --proxy-mode [userspace | iptables | ipvs] If not set, if defaults to iptables.

For services, we set the range in the kube-api-server --service-cluster-ip-range <range> which defaults to 10.0.0.0/24

View rules created by kube proxy

iptables -L -t nat | grep -i servicename

DNS

Domain resolution for services: web-service.namespace.svc.cluster-root-domain i.e db-service.test-ns.svc.cluster.local
For pods , we replace the . with -. This is turned off by default.
Domain resolution for pods: pod-ip-replaced-with-hyphens.namespace.pod.cluster-root-domain i.e 10-244-1-2.test-ns.pod.cluster.local
Coredns watches the cluster for new pods or services and adds a record for them in it's database when they're created and it also creates a service named kube-dns.
Core file for coredns at : etc/coredns/Corefile and it's passed as a config map object
For kubelet, we have the ip of dns server and domain in its config

Example Corefile

{
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

To get the service full domain name:

host service-name

Json Path

Kubernetes docs for json path: Kubernetes Jsonpath
Use the following evaluator for json JsonPath Evaluator

#get field names
k explain deploy.spec --recursive

# Get all car names in an json data is equal to "abc"
# $[] -> return a list
# ?() -> if 
# @ -> Each item in list

$[?(@.name == "abc")]

#Get name and image of first container of the first pod 
kubectl get pods -o=jsonpath='{range .items[0].spec.containers[0]}{.name}{"\n"}{.image}' --sort-by=.name

#Fetch node names 
kubectl get nodes -ojsonpath='{range .items[*].metadata}{.name}

# Sort persisten volumes by capacity
kubectl get pv --sort-by=.spec.capacity.storage

kubectl get pv --sort-by=.spec.capacity.storage -o=custom-columns=NAME:.metadata.name,CAPACITY:.spec.capacity.storage

# get names of user test in config
kubectl config view --kubeconfig=custom-config -o jsonpath="{.contexts[?(@.context.user=='test')].name}"

#Get deployments as per this format:
# DEPLOYMENT   CONTAINER_IMAGE   READY_REPLICAS   NAMESPACE

kubectl get deploy -n admin2406 --sort-by=.metadata.name -o=custom-columns=DEPLOYMENT:.metadata.name,CONTAINER_IMAGE:.spec.template.spec.containers.*.image,READY_REPLICAS:.status.readyReplicas,NAMESPACE:.metadata.namespace

kubectl get svc --sort-by=.metadata.name  -o=custom-columns=NAME:.metadata.name,PORTNAME:.spec.ports.name

k get nodes -ojsonpath='{range .items[*]}InternalIP of {.metadata.name} {.status.addresses[?(@.type=="InternalIP")].address} {end}'

My CKAD Prep

Cosmas Nyairo — Mon, 15 May 2023 16:05:53 +0000

Notes
Pod
Replication Controller and replicaset
Deployments
Namespaces
Services
Config Map
Secrets
Security Context
Service Account
Resource Limits
Taints and Tolerations
Node Selectors
Node Affinity
Multicontainer Pods
Observability
Jobs and Cronjobs
Ingress
Network Policies
Storage
Authentication
Authorization
Admission Controllers
Custom Resource Definition
Custom Controller

Notes

Before we dive into the templates and commands, here are some notes about Kubernetes that you should keep in mind:

Definition file

Cluster is a collection of nodes

Master node has:

api sever (frontend for kube)
etcd (key store for data used to manage cluster)
scheduler (assigns containers to nodes)
controller (bring up containers when they go down)

Worked node has:

kubelet (agent in each node in cluster that ensures containers running on nodes as expected)
runtime (software to run containers in the background i.e docker)

Multi container pods can communicate to each other through localhost

Resources in a namespace can refer to each other by their names

pods have a 1 to 1 relationship with containers
entrypoint in docker -> command in kubernetes
cmd in docker -> args in kubernetes
kube system - resources for internal purposes for kubernetes
kube public - resources to be made available to all users

kubeapiserver path is /etc/kubernetes/manifests/kube-apiserver.yaml
To check settings passed to kubeapi server: ps -aux | grep authorization

To view api resources and their shortnames:

kubectl api-resources

To enable alpha versions: --runtime-config=api/version

Run watch crictl ps to wait for kube api server to come back online

To handle api deprecations:

kubectl convert -f <old-file> --output-version <new-api-version> > <new-file>

Now, let's dive into the templates and commands you can use for resources.

Pod

Single instance of an application (smallest object we can create in k8)

We scale pods up or down

Pod Definition file

kubectl get pods -o wide

kubectl get pods -A

kubectl label pod/<pod-name> <label>=<value>

kubectl get pods,svc

kubectl get pods --no-headers | wc -l

kubectl run <pod-name> --image=<image-name>  -n <namespace> --dry-run=client -o yaml > test.yaml

kubectl set image pod <pod-name> <container-name>=<image>

kubectl get pod <pod-name> -o yaml > pod-definition.yaml

kubectl exec -it podname -- commandtorun

kubectl replace --force -f app.yaml

kubectl apply -f test.yaml

kubeclt edit pod <pod-name>

kubectl explain pods --recursive | less

kubectl explain pods --recursive | grep envFrom -A<number-of-lines>

Pod Design Definition file
Pod labels - used to group objects i.e pods, services, replicasets

kubectl get pods --selector label=value

kubectl get pods --selector label=value,label2=value2

Selectors match labels described i.e match labels on the pod or service to match labels of a pod
Annotation records details for informatory purposes e.g build information, tool used.

Replication Controller and Replicaset

Replication controller is in v1 while replicaset in apps/v1

Replica set uses selector to determine pods to watch and manage even existing pods

kubectl create -f definition.yaml

kubectl get replicationcontroller

kubectl get rs

kubectl delete replicaset <replicaset-name>

kubectl edit replicaset <replicaset-name>

kubectl set image rs <replica-set> <container-name>=<image>

kubectl describe replicaset <replicaset-name>

kubectl apply -f definition.yaml

kubectl replace -f definition.yaml

kubectl scale -replicas=10 -f definition.yaml

kubectl scale --replicas=0 replicaset/<replicaset-name>

Deployments

Deployment Definition file

kubectl create deployment nginx --image=nginx

kubectl scale deploy/webapp --replicas=3

kubectl get deploy

kubectl get deploy -o wide

kubectl delete deployment <deployment -name>

kubectl create deploy redis-deploy --image=redis --replicas=2 -n dev-ns

Deployment strategies:
- Recreate - Remove all applications running on older verision and bring up applications running on newer verision
- Rolling update (default strategy) - Remove a single application and bring up a new one one by one until the newer version is running on all applications
- A new replica set is created under the hood when we do deployment upgrades
- We use record flag to save commands used to create/update deployments
- We use the to revision flag to rollback to a specific revision

kubectl set image deploy/deploymentname <container-name>=<image>

kubectl set image deploy/deploymentname <container-name>=<image> --record

kubectl rollout restart deploy/deploymentname

kubectl rollout status deploy/deploymentname

kubectl rollout history deploy/deploymentname

kubectl rollout undo deploy/deploymentname

kubectl rollout undo deploy/deploymentname --to-revision=1

Namespaces

kubectl create ns <namespace-name>

kubectl config set-context $(kubectl config current-context) -n <namespace-name>

Services

Enable communication between components within the application

Types:

Node Port
- Node Port Definition file
- Map a port on node to a port on the pod
- The node's port Can only be in the range 30000 to 32767
- Node port -> Service -> Target port (Pod's port)
- Node port and service port are not mandatory, if not provided, node port is allocated an available ip in the range 30000 to 32767 while service port is assumed to be same as port
- Acts as loadbalancer if we have multiple pods with the same label, it uses a random algorithm to select which pod to send requests to.
Cluster Ip
- Cluster Ip Definition file
- Service assigned an ip in the cluster and it's used to access the service by other pods in the service.
Load Balancer
- Builds on top of node port and allows balancing of requests to the service to it's target applications

To connect to another service in a different namespace: we use the following syntax:

<service-name>.<namespace>.<svc>.<domain>
test-service.test-ns.svc.cluster.local

kubectl expose resource <resource-name> --type=<type> --port=<port> --target-port=<target-port> --name=<service-name>

kubectl create service <type> <service-name> --tcp=<port>:<port>

Config Map


  kubectl create configmap \
      app-config --from-literal=APP_COLOR=RED \
                 --from-literal=APP_TYPE=AAB \
                 --from-literal=APP_ENV=PROD \

  kubectl create configmap \
      app-config --from-file=app_config.properties

Declarative:

  kubectl create -f test-config-map.yaml

  kubectl get configmaps
  kubectl get cm
  kubectl describe configmap

Secrets

  kubectl create secret generic \
      app-secret --from-literal=DB_HOST=mysql \
                 --from-literal=DB_PORT=1000
                 --from-literal=DB_NAME=test \

  kubectl create secret generic \
      app-secret --from-file=test.env

Declarative

  kubectl create -f test-secret.yaml

  kubectl get secrets

  kubectl get secret <secret-name> -o yaml

  kubectl describe secrets

Security Context

Security Context Definition file

We can add security context on the pod level or the container level. If both are specified, the container level takes precedence.

Cabapilites are supported only on the container level

Service Account

Service Account Definition file

kubectl create serviceaccount test-sa

kubectl get serviceaccount

kubectl describe serviceaccount test-sa

kubectl create token SERVICEACCOUNTNAME

Taints and Tolerations

Taints and Tolerations Definition file

kubectl taint nodes NODENAME app=red:taint-effect

kubectl taint nodes NODENAME app=red:taint-effect-

taint-effects include: NoSchedule|NoExecute|PreferNoSchedule

Node Selectors

Node Selectors Definition file

kubectl label nodes NODENAME type=test

We cannot apply advanced filters e.g not , or

Node Affinity

Node Affinity Definition file

Node affinity types:

requiredDuringSchedulingIgnoredDuringExecution
PreferredDuringSchedulingIgnoredDuringExecution
requiredDuringSchedulingrequiredDuringExecution

	During Scheduling	During Execution
1	Required	Ignored
2	Preferred	Ignored
3	Required	Required

Multicontainer Pods

Multicontainer Pods Definition file
Sidecar containers -> help the main container e.g to logging agent to send logs to log server
Adapter containers -> process data for the main container e.g converts logs to readable format before sending to logging server
Ambassador containers -> proxy requests from the main container e.g send requests to db on main container's behalf
Init Containers Definition file
Init containers -> Process inside init container must finish before other containers start.
If the init container fails, the pod is restarted until the init container succeeds

Observability

Readiness Probe:

Perform test to check if the container is up before marking the container as ready.
For readiness, we can do http calls, tcp calls or run a command that when succesfull, we mark the container as ready

Liveness Probe:

Periodically test if application within container is healthy.
For liveness, we can do http calls, tcp calls or run a command that when they fail, we mark the container as unhealthy and it's restarted
Readiness & Liveness Probe Definition file

Logging:

Show logs

kubectl logs podname

Show live logs

kubectl logs -f podname

For multi container we specify container name

kubectl logs -f podname container-name

Metric Server:

We can have 1 metric server per cluster
Receives metrics from nodes and pods and stores them in memory ( we can't see historical data with metric server)
To install on cluster,we clone the metric server repo and run kubectl create -f repourl

kubectl top node

kubectl top pod

Jobs and Cronjobs

Job - Tasks that can be run and exit after they've finalized

Job Definition file

kubectl get jobs

Cronjob - To run jobs periodically

Cronjob Definition file

kubectl get cronjob

Ingress

Ingress Definition file
Enables users access application through an externally accessible url that we can configure to route to different services in the cluster based on url path while implementing ssl as well
We need to expose it so it can be accessible outside the cluster
Ingress controller:
- Does not come with kubernetes as default. We need to deploy if first.
- Examples are istio, nginx, haproxy
Ingress resources:
- Rules and configs applied to ingress controller to forward traffic to single applications, via paths or via domain name

kubectl get ingress

kubectl create ingress <ingress-name> --rule="host/path=service-name:port"

kubectl create ingress world --rule=world.universe.mine/europe*=europe:80 --rule=world.universe.mine/asia*=asia:80  --class=nginx -n world

Network Policies

Allow and deny rules configured on the pod
Network Policies Definition file

kubectl get netpol

Storage

Volumes
- Volume Definition file
Persistent Volumes
- Cluster wide pool of storage volumes to be used by applications on the cluster
- Applications can then request storage from the pool to use
- Persistent Volumes Definition file
- Persistent Volume in a Pod Definition file
Persistent Volume Claim
- Users create persistent volume claims to use the storage in the persistent volume
- Kubernetes binds pvc to pv based on request and properties on the volume
- Persistent Volume Claim Definition file

  kubeclt delete pvc <pvc-claim>

Storage Class
- We define a provisioner to automatically provision storaage which can be used by pods.
- If we specify the storage class, we don't need to specify the persistent volume as it would be created automatically when storage class is created.
- Storage Class Definition file
- Persistent Volume Claim Storage Class Definition file

Authentication

All user access is managed by the api server

We can store user credentials as:

Static Password File:

    # Add this to kubeapi server service or pod definition file
    --basic-auth-file=user-credentials.csv

    curl -v -k <api-url> -u "user:password"

Static Token File:

   # Add this to kubeapi server service or pod definition file
   --token-auth-file=user-credentials.csv

    curl -v -k <api-url> --header "Authorization: Bearer <TOKEN>"

We can use kubeconfig to manage which clusters we can access

Kubeconfig Definition file

kubectl config view --kubeconfig=my-custom-file

kubectl config use-context developer-development-playground

kubectl config get-context developer-development-playground

Authorization

Authorization modes:

Node authorizer -> handles node requests (user should have name prefixed system node)
Attribute based authotization -> Assosiate users/group of users with a set of permissions(difficult to manage)
Role bases access control -> We define roles and associate users with specific roles
Webhook -> Outsource authorization to 3rd party tools
AlwaysAllow -> Allows all requests without doing authorization checks (default)
AlwaysDeny -> Denys all requests

On kubeapiserver, we specify modes to use --authorization-mode=Node,RBAC,Webhook

Role based access control:

kubectl get roles

kubectl create role test --verb=list,create --resource=pods

kubectl describe role <role-name>

kubectl get rolebindings

kubectl create rolebinding test-rb --role=test --user=user1 --group=group1

kubectl describe rolebindings <rolebindings-name>

kubectl auth can-i create deploy

kubectl auth can-i create deploy --as dev-user --namespace dev

kubectl api-resources --namespaced=false

We can create roles scoped on clusters. We can also create cluster roles on namespace scoped resources

Cluster Role based access control:

kubectl create clusterrole test --verb=* --resource=*

kubectl create clusterrolebinding test-rb --clusterrole=test --user=user --group=group

Admission Controllers

Implement security measures to enforce how a cluster is used.
It can validate, change or reject requests from users
It can also perform operations in the backend

kube-apiserver -h | grep enable-admission-plugins

On kubeapiserver, to enable an admission controller we update the --enable-admission-plugins=NodeRestriction,NamespaceLifecycle
On kubeapiserver, to disable an admission controller we update the --disable-admission-plugins=DefaultStorageClass

We can create custom admission contollers:

We use the Mutating and Validating webhooks that we configure to a server hosting our admission webhook service.
If a request is made, it goes sends an admission review object to admission webhook server that responds whith an admission review object of whether the result is allowed or not
We then deploy our admission webhook server
We then configure to reach out to the service and validate or mutate requests by creating a validating configuration object
- Validating Configuration Definition file

Custom Resource Definition

An extension of the Kubernetes API that isn't available in the Kubernetes installation

Custom Resource Definition file

Custom Controller

Process or code running in a loop and monitoring the kubernetes cluster and listening to events of specific objects

We build the custom controller then provide kubeconfig file the controller would need to authenticate to the kubernetes api

Thanks!

Creating & Managing Aws Rest Api Gateway Using Terraform

Cosmas Nyairo — Mon, 09 Jan 2023 16:44:08 +0000

We will be creating and managing an AWS rest api gateway using terraform

Introduction
Prerequisites
Rest Api
Rest Api Body
Open API server
Open API paths
Method response
Method details
Terraform File structure

Introduction

We can create resources on cloud either manually or using automation tools.

Lets take an example of the following api endpoint our service is exposing either through lambda or the vpc link:



 api/v1/test/get_tests -> an enpoint with POST AND GET requests

If we were to create then above method manually; we would have to the following:

Create the rest api if one didnt exist
Create the api,v1, test then the get_tests resources
Create the GET method then and add the needed headers, add parameters the api may need, add authorization to the api and then repeat the same for the POST method.

This would be cumbersome if we had many api endpoints to create and managing the api endpoints would be very hard.

In this post:

We will focus on creating a simple AWS rest api which we will manage using terraform (an Infrastructure as Code tool).
We will be using the openapi specification for our apis
This approach can be inegrated with a CI/CD tool i.e jenkins or gitlab ci or github actions to enable further automation of our rest api endpoint

Prerequisites:

A postman collection containing our api endpoints (swagger api documentation would work as well)
An AWS account
Terraform (The version in use is Terraform v1.2.6)
We will focus on the rest apis where the underlying service is exposed using the vpc link

Rest api

For our rest api, we will need to provide the name, description and the endpoint type.



resource "aws_api_gateway_rest_api" "test_rest_api"{
  name = "<<REST API NAME>>"
  endpoint_configuration {
    types = ["REGIONAL"]
  }
  description = "<<REST API DESCRIPTION>>"
  body        = data.template_file.testfile.rendered
}

Rest api body

For the body we will use the code snippet below that utilizes terraform template files:



data "template_file" "testfile"{
  template = file("./test.yaml")
  vars = {
    methodresponses = var.methodresponses
    methoddetails   = var.methoddetails
  }
}

Open API server

The servers section contains the custom domain name we would use to expose our api using.

Users would be able to access the service by sending api requests to the "{domainName}/{basePath}" endpoint

Open API paths

The paths section contains the paths for our api endpoints. We only need to specify the following:

Api endpoint e.g /api/v1/test/get_tests
The absolute path of the api endpoint being exposed via the vpc link ("https://$${stageVariables.url}/api/v1/test/get_tests")
- The first part of the url is the url of the underlying domain exposing the service i.e "https://$${stageVariables.url}/.
- The latter part is the actual path the service expects to expose the endpoint i.e "/api/v1/test/get_tests"
The type of the http method e.g POST

Method response

The method reponses contains a of method reponse e.g 200 and the response headers for the reponse.



variable "methodresponses"{
  default = <<EOT
responses:
        "200":
          description: "200 response"
          headers:
            Access-Control-Allow-Headers:
              schema:
                type: "string"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Empty"
EOT
}

Method details

The method detail are as per the template below. We can add any headers the method needs under the response parameters section as shown.



variable "methoddetails"{
  default = <<EOT
responses:
  default:
    statusCode: "200"
    responseParameters:
      method.response.header.Access-Control-Allow-Origin: "<<VALUE>>"
passthroughBehavior: "when_no_match"
connectionType: "VPC_LINK"
type: "http_proxy"
EOT
}

Terraform File structure

Our main.tf file will contain the following structure:



resource "aws_api_gateway_rest_api" "test_rest_api"{
  name = "<<REST API NAME>>"
  endpoint_configuration {
    types = ["REGIONAL"]
  }
  description = "<<REST API DESCRIPTION>>"
  body        = data.template_file.testfile.rendered
} 

data "template_file" "testfile"{
  template = file("./test.yaml")
  vars = {
    methodresponses = var.methodresponses
    methoddetails   = var.methoddetails
  }
}

Our locals.tf will contain the following structure:



variable "methodresponses"{
  default = <<EOT
responses:
        "200":
          description: "200 response"
          headers:
            Access-Control-Allow-Headers:
              schema:
                type: "string"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Empty"
EOT
}

variable "methoddetails"{
  default = <<EOT
responses:
  default:
    statusCode: "200"
    responseParameters:
      method.response.header.Access-Control-Allow-Origin: "<<VALUE>>"
passthroughBehavior: "when_no_match"
connectionType: "VPC_LINK"
type: "http_proxy"
EOT
}

Our test.yaml file will contain the following openapi specification:



openapi: "3.0.1"
servers:
  - url: "{domainName}/{basePath}"
    variables:
      domainName:
        default: "https://test.com"
      basePath:
        default: "/testpath"
paths:
  /api/v1/test/get_tests:
    post:
      ${methodresponses}
      x-amazon-apigateway-integration:
        uri: "https://$${stageVariables.url}/api/v1/test/get_tests"
        connectionId: "$${stageVariables.vpcLink}"
        httpMethod: "POST"
        ${methoddetails}
    get:
      ${methodresponses}
      x-amazon-apigateway-integration:
        uri: "https://$${stageVariables.url}/api/v1/test/get_tests"
        connectionId: "$${stageVariables.vpcLink}"
        httpMethod: "GET"
        ${methoddetails}
  /api/v1/test/tests:
    get:
      ${methodresponses}
      x-amazon-apigateway-integration:
        uri: "https://$${stageVariables.url}//api/v1/test/tests"
        connectionId: "$${stageVariables.vpcLink}"
        httpMethod: "GET"
        ${methoddetails}

To manage/add a new api endpoint we only need to update/change the test.yaml file.

We only need to do terraform plan and terraform apply for our api to be created.

DEV Community: Cosmas Nyairo

Gateway VPC Endpoints on AWS

Gateway VPC Endpoints Example:

Certified Kubernetes Administrator (CKA) Prep.

Table of Contents

Notes

Scheduling

Manual scheduling

Labels and Selectors

Taints and Tolerations

Node Selector

Node affinity

Resource Limits

Daemon Sets

Static Pods

Multiple Schedulers

Scheduler Profiles

Monitoring and Logging

Os Upgrades

Security

Authentication

TLS Certificates

Kubeconfig

Authorization

Network Policy

Storage

Networking

Network configuration - Cluster Nodes

Pod Networking

Service Networking

DNS

Json Path

My CKAD Prep

Table of Contents

Notes

Pod

Replication Controller and Replicaset

Deployments

Namespaces

Services

Config Map

Secrets

Security Context

Service Account

Taints and Tolerations

Node Selectors

Node Affinity

Multicontainer Pods

Observability

Jobs and Cronjobs

Ingress

Network Policies

Storage

Authentication

Authorization

Admission Controllers

Custom Resource Definition

Custom Controller

Creating & Managing Aws Rest Api Gateway Using Terraform

Table of Contents

Introduction

Prerequisites:

Rest api

Rest api body

Open API server

Open API paths

Method response

Method details

Terraform File structure