DEV Community: Kaito Ii

Monitor Kubernetes Events as Logs with Fluent Bit

Kaito Ii — Thu, 07 Sep 2023 09:37:45 +0000

Kubernetes event is report of an event happening within the cluster. It tracks any changes made to the cluster which is a good piece of information for debugging.
However, the retention time is limited.

Fluentbit

Fluent Bit is a fast and lightweight Telemetry agent. We will be using Kubernetes Events input plugin to process the Kubernetes event as log.

Monitor Kubernetes Event

First we will install Fluentbit using Helm. We will assume you already have a Kubernetes cluster handy.

We will deploy Fluentbit with a minimal configuration as shown below. Save this as values.yaml.

rbac:
  eventsAccess: true

config:
  inputs: |
    [INPUT]
        Name kubernetes_events
        Tag k8s_events.*
        kube_url https://kubernetes.default.svc

  outputs: |
    [OUTPUT]
        Name stdout
        Match k8s_events.*

We will need rbac.eventsAccess: true to give Fluentbit access to Kubernetes Event resource. In config.inputs, we define the input of Fluentbit which will be Kubernetes event in this case. And in config.outputs, we define where the output of the incoming Kubernetes event goes to.

$ helm repo add fluent https://fluent.github.io/helm-charts

$ helm repo update

$ helm upgrade --install fluent-bit fluent/fluent-bit -f values.yaml

Let's make sure that our Fluentbit is running.

 kubectl get po
NAME               READY   STATUS    RESTARTS        AGE
fluent-bit-4dm2l   1/1     Running   0               27h

We got the fluentbit running so let's create a sample pod to create some Kubernetes events.

$ kubectl run nginx --image=nginx -- sleep 1000

And if we take a look at the fluent bit log, we can see the events related to nginx pod we just created.

$ kubectl logs ds/fluent-bit
[1] k8s_events.*: [[1694078558.000000000, {}], {"metadata"=>{"name"=>"nginx.17829379194ff602", "namespace"=>"default", "uid"=>"08314b94-2b02-4485-8a86-65cb07f1b3e1", "resourceVersion"=>"164604", "creationTimestamp"=>"2023-09-07T09:22:38Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:38Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Pulling", "message"=>"Pulling image "nginx"", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:38Z", "lastTimestamp"=>"2023-09-07T09:22:38Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]
[0] k8s_events.*: [[1694078563.000000000, {}], {"metadata"=>{"name"=>"nginx.1782937a6e3d22e3", "namespace"=>"default", "uid"=>"2fe002bc-c1c5-47b5-b057-a0764602893a", "resourceVersion"=>"164611", "creationTimestamp"=>"2023-09-07T09:22:43Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:43Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Pulled", "message"=>"Successfully pulled image "nginx" in 5.71974592s (5.71977817s including waiting)", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:43Z", "lastTimestamp"=>"2023-09-07T09:22:43Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]
[1] k8s_events.*: [[1694078563.000000000, {}], {"metadata"=>{"name"=>"nginx.1782937a6ea78c6c", "namespace"=>"default", "uid"=>"0058a999-0966-4b66-84c8-3361622fc7a8", "resourceVersion"=>"164612", "creationTimestamp"=>"2023-09-07T09:22:43Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:43Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Created", "message"=>"Created container nginx", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:43Z", "lastTimestamp"=>"2023-09-07T09:22:43Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]
[2] k8s_events.*: [[1694078563.000000000, {}], {"metadata"=>{"name"=>"nginx.1782937a735969d5", "namespace"=>"default", "uid"=>"8830ae74-2f2c-4d5f-92a8-6f48aff7de59", "resourceVersion"=>"164615", "creationTimestamp"=>"2023-09-07T09:22:43Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:43Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Started", "message"=>"Started container nginx", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:43Z", "lastTimestamp"=>"2023-09-07T09:22:43Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]

If we take a look at the Kubernetes event with kubectl, we can see the same event which we saw in the Fluentbit log for long term storage.

$ kubectl get events --field-selector involvedObject.name=nginx                                           [18:31:21]
LAST SEEN   TYPE     REASON      OBJECT      MESSAGE
8m45s       Normal   Scheduled   pod/nginx   Successfully assigned default/nginx to kind-control-plane
8m44s       Normal   Pulling     pod/nginx   Pulling image "nginx"
8m39s       Normal   Pulled      pod/nginx   Successfully pulled image "nginx" in 5.71974592s (5.71977817s including waiting)
8m39s       Normal   Created     pod/nginx   Created container nginx
8m39s       Normal   Started     pod/nginx   Started container nginx

We tested out the Kubernetes event input plugin with a simple configuration. From here we can customize the fluentbit configuration to send the logs to third party storages outside of Kubernetes cluster .

Reference

Remote writing your Prometheus metrics to Grafana Cloud

Kaito Ii — Fri, 18 Mar 2022 04:23:53 +0000

Setup Grafana Cloud

Create API Key for Grafana Cloud

We will create an API Key to allow our local Prometheus to remote write to Grafana Cloud.

Head over to your Grafana Cloud Portal and select Send Metrics on Prometheus.

If you scroll above, you should see the section for API Key. Click on Generate now and create an API Key with the Role MetricsPublisher.
Copy the Prometheus config and save it locally as values.yaml as we will use it to install our Prometheus Helm Chart. The url and username should be unique for every user.
The password in both snippet should be filled with your API key.

Local Setup

I used Rancher Desktop but most of Kubernetes distribution should work similarly. We will deploy Prometheus using kube-prometheus-stack with Helm.

Add Helm repository ```

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update


Before installing Prometheus, we will tweak the values.yaml we created earlier. We will not directly paste the username and password(API Key) to the file. Instead, we will create a secret and reference that from Prometheus Config.

$ kubectl create secret generic grafanacloudsecret \
--from-literal=username= \
--from-literal=password=''

Now, edit the values.yaml. Here is the edited version. Notice that the username and password is referencing the secret. Also I have added extra parameter which will attach extra labels to the metrics sent from this local cluster.

```yaml


prometheus:
  prometheusSpec:
    remoteWrite:
    - url: "https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push"
      basicAuth:
          username:
            name: grafanacloudsecret
            key: username
          password:
            name: grafanacloudsecret
            key: password
    replicaExternalLabelName: "__replica__"
    externalLabels: {cluster: "mylocalcluster"}

Now, let's install our Prometheus.



$ helm install prometheus prometheus-community/kube-prometheus-stack -n prometheus --create-namespace -f values.yaml

Check the metrics on Grafana Cloud

Go back to your Grafana Cloud and open the Grafana Cloud Billing/Usage. If you wait awhile, we should see some metrics being ingested in Grafana Cloud.

On the left pane, Go to Explore. In the Metrics browser, type in container_memory_usage_bytes. We can see the memory usage in my local Kubernetes cluster. Notice that there is an extra label named cluster: "mylocalcluster" which we added in our values.yaml. This can be used as an identifier for where this data is coming from.

Reference

WASM workload on AKS

Kaito Ii — Thu, 21 Oct 2021 13:36:58 +0000

Public preview of AKS supporting WASI (WebAssembly System Interface) workload in Kubernetes is out. And I thought I would try it out.

https://azure.microsoft.com/en-us/updates/public-preview-aks-support-for-webassembly-system-interface-wasi-workloads/

Register WasmNodePoolPreview feature

$ az feature register --namespace "Microsoft.ContainerService" --name "WasmNodePoolPreview"
{
  "id": "/subscriptions/<YOUR SUBSCRIPTION ID>/providers/Microsoft.Features/providers/Microsoft.ContainerService/features/WasmNodePoolPreview",
  "name": "Microsoft.ContainerService/WasmNodePoolPreview",
  "properties": {
    "state": "Registering"
  },
  "type": "Microsoft.Features/providers/features"
}

$ az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/WasmNodePoolPreview')].{Name:name,State:properties.state}"
Name                                            State
----------------------------------------------  -----------
Microsoft.ContainerService/WasmNodePoolPreview  Registering

$ az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/WasmNodePoolPreview')].{Name:name,State:properties.state}"
Name                                            State
----------------------------------------------  ----------
Microsoft.ContainerService/WasmNodePoolPreview  Registered

Install aks-preview extension

$ az extension add --name aks-preview
$ az extension update --name aks-preview

Add WASM/WASI node pool to AKS cluster

# Create resource group
$ az group create --name wasmRG -l westus
# Create AKS cluster
$ az aks create --resource-group wasmRG --name myAKSCluster
# Add WASM/WASI node pool
$ az aks nodepool add --resource-group wasmRG --cluster-name myAKSCluster --name mywasipool --node-count 1 --workload-runtime wasmwasi
# Check the workloadRuntime
$ az aks nodepool show -g wasmRG --cluster-name myAKSCluster -n mywasipool | jq '.workloadRuntime'
"WasmWasi"

# Get credential for AKS cluster
$ az aks get-credentials -n myakscluster -g wasmRG

$ kubectl get nodes
NAME                                 STATUS   ROLES   AGE   VERSION
aks-mywasipool-11259355-vmss000000   Ready    agent   76m   1.0.0-alpha.1
aks-nodepool1-11259355-vmss000000    Ready    agent   80m   v1.20.9
aks-nodepool1-11259355-vmss000001    Ready    agent   80m   v1.20.9
aks-nodepool1-11259355-vmss000002    Ready    agent   80m   v1.20.9

# Get labels on node
$ kubectl get node aks-mywasipool-11259355-vmss000000 -o jsonpath='{.metadata.labels}' | jq
{
  "agentpool": "mywasipool",
  "beta.kubernetes.io/arch": "wasm32-wagi",
  "beta.kubernetes.io/os": "wasm32-wagi",
  "kubernetes.azure.com/agentpool": "mywasipool",
  "kubernetes.azure.com/cluster": "MC_wasmRG_myAKSCluster_westus",
  "kubernetes.azure.com/mode": "user",
  "kubernetes.azure.com/node-image-version": "AKSUbuntu-1804gen2containerd-2021.10.02",
  "kubernetes.azure.com/os-sku": "Ubuntu",
  "kubernetes.azure.com/role": "agent",
  "kubernetes.azure.com/storageprofile": "managed",
  "kubernetes.azure.com/storagetier": "Premium_LRS",
  "kubernetes.io/arch": "wasm32-wagi",
  "kubernetes.io/hostname": "aks-mywasipool-11259355-vmss000000",
  "kubernetes.io/os": "wasm32-wagi",
  "kubernetes.io/role": "agent",
  "node-role.kubernetes.io/agent": "",
  "storageprofile": "managed",
  "storagetier": "Premium_LRS",
  "type": "krustlet"
}

# Get Taints defined in node
$ kubectl get node aks-mywasipool-11259355-vmss000000 -o jsonpath='{.spec.taints}' | jq
[
  {
    "effect": "NoSchedule",
    "key": "kubernetes.io/arch",
    "value": "wasm32-wagi"
  },
  {
    "effect": "NoExecute",
    "key": "kubernetes.io/arch",
    "value": "wasm32-wagi"
  }
]

label "type": "krustlet" shows that the node runs krustlet instead of kubelet
label "kubernetes.io/arch": "wasm32-wagi" shows the node architecture
taint on the wasi node pool will only schedule/execute pods with toleration of wasm32-wagi

Run WASM/WASI Workload

Create the following yaml as wasi-example.yaml
- nodeSelector will choose the node with certain label (i.e. node with kubernetes.io/arch: wasm32-wagi)
- toleration will tolerate the taint in the node (i.e. wasm32-wagi NoSchedule and NoExecute)

apiVersion: v1
kind: Pod
metadata:
  name: krustlet-wagi-demo
  labels:
    app: krustlet-wagi-demo
  annotations:
    alpha.wagi.krustlet.dev/default-host: "0.0.0.0:3001"
    alpha.wagi.krustlet.dev/modules: |
      {
        "krustlet-wagi-demo-http-example": {"route": "/http-example", "allowed_hosts": ["https://api.brigade.sh"]},
        "krustlet-wagi-demo-hello": {"route": "/hello/..."},
        "krustlet-wagi-demo-error": {"route": "/error"},
        "krustlet-wagi-demo-log": {"route": "/log"},
        "krustlet-wagi-demo-index": {"route": "/"}
      }
spec:
  hostNetwork: true
  nodeSelector:
    kubernetes.io/arch: wasm32-wagi
  containers:
    - image: webassembly.azurecr.io/krustlet-wagi-demo-http-example:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-http-example
    - image: webassembly.azurecr.io/krustlet-wagi-demo-hello:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-hello
    - image: webassembly.azurecr.io/krustlet-wagi-demo-index:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-index
    - image: webassembly.azurecr.io/krustlet-wagi-demo-error:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-error
    - image: webassembly.azurecr.io/krustlet-wagi-demo-log:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-log
  tolerations:
    - key: "node.kubernetes.io/network-unavailable"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wagi"
      effect: "NoExecute"
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wagi"
      effect: "NoSchedule"

$ kubectl apply -f wasi-example.yaml

$ kubectl get po -o wide
NAME                 READY   STATUS       RESTARTS   AGE   IP       NODE                                 NOMINATED NODE   READINESS GATES
krustlet-wagi-demo   0/5     Registered   0          78s   <none>   aks-mywasipool-11259355-vmss000000   <none>           <none>

We can see that the pods has been scheduled on wasipool node

# Get Internal IP for WASI Node
$ WASINODE_IP=$(kubectl get nodes aks-mywasipool-11259355-vmss000000 -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')

# Create values.yaml for nginx chart
$ cat << EOF > values.yaml
serverBlock: |-
  server {
    listen 0.0.0.0:8080;
    location / {
            proxy_pass http://$WASINODE_IP:3001;
    }
  }
EOF

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm repo update
$ helm install hello-wasi bitnami/nginx -f values.yaml

$ kubectl get service 
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
hello-wasi-nginx   LoadBalancer   10.0.163.230   40.83.129.0   80:32513/TCP   3m11s
kubernetes         ClusterIP      10.0.0.1       <none>        443/TCP        152m

# Get the external ip for the hello-wasi-nginx service
$ EXTERNAL_IP=$(kubectl get service hello-wasi-nginx -o jsonpath='{.status.loadBalancer.ingress[].ip}')

$ curl $EXTERNAL_IP/hello
hello world

Create my own WASM workload with Rust

Prerequisite: Rust, wasmtime
Set up the Rust environment

$ mkdir aks-wasm; cd aks-wasm
$ cargo init 
$ rustup target add wasm32-wasi

Create a simple app for wasm (src/main.rs)

fn main() {
    println!("Content-Type: text/plain\n");
    println!("Hello from wasm!");
}

Build the app and run it locally

$ cargo build --release --target wasm32-wasi
$ wasmtime target/wasm32-wasi/release/aks-wasm.wasm
Content-Type: text/plain

Hello from wasm!

Upload the WASM module to Azure Container Registry

Prerequisite: wasm-to-oci
Setup Azure ContainerRegistry

# You may have to choose a different name for the registry 
$ az acr create -n MyWASMRegistry -g wasmRG --sku Standard

# login to Azure Container Registry. Make sure Docker is running
$ az acr login -n MyWASMRegistry

# Enable anonymous pull to allow pulling the WASM module from WASI node
$ az acr update --name myregistry --anonymous-pull-enabled

Push the WASM module to Azure Container Registry


# Push the WASM module to Azure Container Registry
$ wasm-to-oci push target/wasm32-wasi/release/aks-wasm.wasm mywasmregistry.azurecr.io/mywasm:1.0.0 

$ az acr repository list -n MyWASMRegistry
[
  "mywasm"
]

# Make sure the WASM module has been uploaded
$ az acr repository show -n MyWASMRegistry --image mywasm:1.0.0                                           [18:27:29]
{
  "changeableAttributes": {
    "deleteEnabled": true,
    "listEnabled": true,
    "readEnabled": true,
    "writeEnabled": true
  },
  "createdTime": "2021-10-21T09:21:15.8964565Z",
  "digest": "sha256:3cade5d310ece611b4d87df55c64e4e6d99ae08d052cfaef2e699df56db90fef",
  "lastUpdateTime": "2021-10-21T09:21:15.8964565Z",
  "name": "1.0.0",
  "signed": false
}

Deploy my WASM module

Update wasi-example.yaml by adding the wasm module we created earlier

apiVersion: v1
kind: Pod
metadata:
  name: krustlet-wagi-demo
  labels:
    app: krustlet-wagi-demo
  annotations:
    alpha.wagi.krustlet.dev/default-host: "0.0.0.0:3001"
    alpha.wagi.krustlet.dev/modules: |
      {
        "krustlet-wagi-demo-http-example": {"route": "/http-example", "allowed_hosts": ["https://api.brigade.sh"]},
        "krustlet-wagi-demo-hello": {"route": "/hello/..."},
        "krustlet-wagi-demo-error": {"route": "/error"},
        "krustlet-wagi-demo-log": {"route": "/log"},
        "krustlet-wagi-demo-index": {"route": "/"},
        "krustlet-wagi-mywasm": {"route": "/mywasm"}
      }
spec:
  hostNetwork: true
  nodeSelector:
    kubernetes.io/arch: wasm32-wagi
  containers:
    - image: webassembly.azurecr.io/krustlet-wagi-demo-http-example:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-http-example
    - image: webassembly.azurecr.io/krustlet-wagi-demo-hello:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-hello
    - image: webassembly.azurecr.io/krustlet-wagi-demo-index:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-index
    - image: webassembly.azurecr.io/krustlet-wagi-demo-error:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-error
    - image: webassembly.azurecr.io/krustlet-wagi-demo-log:v1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-demo-log
    - image: mywasmregistry.azurecr.io/mywasm:1.0.0
      imagePullPolicy: Always
      name: krustlet-wagi-mywasm
  tolerations:
    - key: "node.kubernetes.io/network-unavailable"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wagi"
      effect: "NoExecute"
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wagi"
      effect: "NoSchedule"

Restart the AKS cluster since the WASM module does not get updated for some reason.

# Delete current WASM workload
$ kubectl delete -f wasi-example.yaml

# Restart AKS cluster
$ az aks stop -n myakscluster -g wasmRG && az aks start -n myakscluster -g wasmRG

# Redeploy WASM workload
$ kubectl applt -f wasi-example.yaml

$ EXTERNAL_IP=$(kubectl get service hello-wasi-nginx -o jsonpath='{.status.loadBalancer.ingress[].ip}')

$ curl $EXTERNAL_IP/mywasm
Hello from wasm!

Clean up

$ helm delete hello-wasi
$ kubectl delete -f wasi-example.yaml
$ az aks nodepool delete --name mywasipool -g wasmRG --cluster-name myakscluster

Reference

Notes on CRL

Kaito Ii — Wed, 04 Aug 2021 14:16:48 +0000

CRL: Certificate Revocation List

a list of revoked certificates published by a CA or a delegated CRL issuer
a mechanism for canceling client-side certificate
when issuing a certificate, the CA includes CRL infromation for the certificate in the certificate itself
may or may not CDP information within the certificate
the system compares the user's certificate against apporpriate CRL during authentication
- It determined to be valid, the system caches certificate attributes and applies
- if it determined that certificate is invalid, it cannot contact apporpriate CRL, or if CRL is expired, it denies the user access

CDP: CRL Distribution Point

location on an LDAP directory server or Web server where CA publishes CRLs
the system periodically contacts CDP to get an update of CRL
the system downloads cRL information from CDP at the interval specified in the CRL, at interval that you specify during CRL configuration, and when manually download the CRL
Use any of the following methods to notify the system of a certificate's CDO location
- Specify the CDP in CA certificate
  - location of CDP may be included within the certificaate
- Specify the CDP in client certificate
  - location of CDP may be included within the certificaate
- Require administrator to mannually enter the CDP location
  - if neither CA or client certificate include the CDP llocation, you must manually specify how to download the CRL objecy

Command to easily maintain rust dependencies with command line

Kaito Ii — Sun, 21 Mar 2021 04:11:24 +0000

Cargo edit

Cargo edit extends cargo by adding the following subcommands.
As you can guess by their names, the subcommands modify Cargo.toml from the command line.

cargo add
cargo rm
cargo upgrade

Install

$ cargo install cargo-edit

cargo add

$ cargo add image
Updating 'https://github.com/rust-lang/crates.io-index' index
Adding image v0.23.14 to dependencies

$ cat Cargo.toml                                                                                                          [dependencies]
image = "0.23.14"

Add dependencies with a specific version.

$ cargo add image --vers=0.20                                                                                                  Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding image v0.20 to dependencies

 ii@ii-pro  ~/t/r/image   master ⁝ ?
 $ cat Cargo.toml                                                                                                           [dependencies]
image = "0.20"

cargo upgrade

$ cargo upgrade image                                                                                                      
    Updating 'https://github.com/rust-lang/crates.io-index' index
image:
    Upgrading image v0.20 -> v0.23.14

cargo rm

$ cargo rm image
Removing image from dependencies

$ cat Cargo.toml

Deploy Prometheus monitoring stack to Kubernetes with a single Helm Chart

Kaito Ii — Mon, 01 Mar 2021 09:30:55 +0000

kube-prometheus-stack

kube-prometheus-stack is a collection of Kubernetes manifests including the follow

Prometheus operator
Prometheus
Alertmanager
Prometheus node-exporter
Prometheus Adapter
kube-state-metrics
Grafana
pre-configured to collect metrics from all Kubernetes component
delivers a default set of dashboards and alerting rules

Prerequiste

Kubernetes
Helm

Install

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

$ kubectl create ns prom
$ helm install prometheus prometheus-community/kube-prometheus-stack -n prom

$ kubectl get all -n prom
NAME                                                         READY   STATUS    RESTARTS   AGE
pod/alertmanager-prom-kube-prometheus-stack-alertmanager-0   2/2     Running   0          2m14s
pod/prom-grafana-6c578f9954-jd4nc                            2/2     Running   0          2m15s
pod/prom-kube-prometheus-stack-operator-598f86d8d7-759tf     1/1     Running   0          2m15s
pod/prom-kube-state-metrics-85d7ddf577-bmnzz                 1/1     Running   0          2m15s
pod/prom-prometheus-node-exporter-6kf8n                      1/1     Running   0          2m16s
pod/prometheus-prom-kube-prometheus-stack-prometheus-0       2/2     Running   1          2m13s

NAME                                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/alertmanager-operated                     ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   2m14s
service/prom-grafana                              ClusterIP   10.107.46.206    <none>        80/TCP                       2m16s
service/prom-kube-prometheus-stack-alertmanager   ClusterIP   10.107.9.211     <none>        9093/TCP                     2m16s
service/prom-kube-prometheus-stack-operator       ClusterIP   10.104.10.237    <none>        443/TCP                      2m16s
service/prom-kube-prometheus-stack-prometheus     ClusterIP   10.110.117.167   <none>        9090/TCP                     2m16s
service/prom-kube-state-metrics                   ClusterIP   10.106.62.213    <none>        8080/TCP                     2m16s
service/prom-prometheus-node-exporter             ClusterIP   10.96.109.34     <none>        9100/TCP                     2m16s
service/prometheus-operated                       ClusterIP   None             <none>        9090/TCP                     2m13s

NAME                                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prom-prometheus-node-exporter   1         1         1       1            1           <none>          2m16s

NAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prom-grafana                          1/1     1            1           2m16s
deployment.apps/prom-kube-prometheus-stack-operator   1/1     1            1           2m16s
deployment.apps/prom-kube-state-metrics               1/1     1            1           2m16s

NAME                                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/prom-grafana-6c578f9954                          1         1         1       2m16s
replicaset.apps/prom-kube-prometheus-stack-operator-598f86d8d7   1         1         1       2m16s
replicaset.apps/prom-kube-state-metrics-85d7ddf577               1         1         1       2m16s

NAME                                                                    READY   AGE
statefulset.apps/alertmanager-prom-kube-prometheus-stack-alertmanager   1/1     2m14s
statefulset.apps/prometheus-prom-kube-prometheus-stack-prometheus       1/1     2m13s

Access Prometheus Dashboard

All services are defined as ClusterIP in default configuration. To access, we are going to use port-forward. However we can edit the service or edit the value upon deployment to use NodePort or Ingress.

$ kubectl port-forward -n prom prometheus-prom-kube-prometheus-stack-prometheus-0 9090

Access Grafana Dashboard

default user/password is admin/prom-operator

$ kubectl port-forward -n prom prom-grafana-6c578f9954-rjdmk 3000

Uninstall

$ helm uninstall prom -n prom

Remove CRDs
$ kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
$ kubectl delete crd alertmanagers.monitoring.coreos.com
$ kubectl delete crd podmonitors.monitoring.coreos.com
$ kubectl delete crd probes.monitoring.coreos.com
$ kubectl delete crd prometheuses.monitoring.coreos.com
$ kubectl delete crd prometheusrules.monitoring.coreos.com
$ kubectl delete crd servicemonitors.monitoring.coreos.com
$ kubectl delete crd thanosrulers.monitoring.coreos.com

DEV Community: Kaito Ii

Monitor Kubernetes Events as Logs with Fluent Bit

Fluentbit

Monitor Kubernetes Event

Reference

Remote writing your Prometheus metrics to Grafana Cloud

Setup Grafana Cloud

Create API Key for Grafana Cloud

Local Setup

Check the metrics on Grafana Cloud

Reference

WASM workload on AKS

TOC

Register WasmNodePoolPreview feature

Install aks-preview extension

Add WASM/WASI node pool to AKS cluster

Run WASM/WASI Workload

Create my own WASM workload with Rust

Upload the WASM module to Azure Container Registry

Deploy my WASM module

Clean up

Reference

Notes on CRL

CRL: Certificate Revocation List

CDP: CRL Distribution Point

Command to easily maintain rust dependencies with command line

Cargo edit

Install

cargo add

cargo upgrade

cargo rm

Deploy Prometheus monitoring stack to Kubernetes with a single Helm Chart

kube-prometheus-stack

Prerequiste

Install

Access Prometheus Dashboard

Access Grafana Dashboard

Uninstall