DEV Community: Cheedge Lee

CKS Notes - Audit log

Cheedge Lee — Fri, 28 Nov 2025 21:14:12 +0000

This is a quick notes and summary from office doc.

this is a quick look at log audit.
more detail about the log audit, pls refer to the official doc about audit
and also can check my previous article: kubernets related logs & configurations which shows clear log file paths and the config related files and their locations.

Audit workflow

edit the audit-policy.yaml:

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"

add flags to /etc/kubernetes/manifests/kube-apiserver.yaml

spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=8
    - --audit-log-maxsize=9
    - --audit-log-maxbackup=3

--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log
--audit-log-maxage=8 : max number of days
--audit-log-maxsize=9 : max file size in megabytes before rotate
--audit-log-maxbackup=3 : copy of audit log files

edit the volumeMounts

volumeMounts:
  - mountPath: /etc/kubernetes/audit-policy.yaml
    name: audit
    readOnly: true
  - mountPath: /var/log/kubernetes/audit/
    name: audit-log
    readOnly: false

edit the hostPath corresponding with --audit-policy-file and --audit-log-path

volumes:
- name: audit
  hostPath:
    path: /etc/kubernetes/audit-policy.yaml
    type: File

- name: audit-log
  hostPath:
    path: /var/log/kubernetes/audit/
    type: DirectoryOrCreate

and then wait for some time and we can check the logs on path /var/log/kubernetes/audit/.

Notice:

in above Volume and VolumeMount, we see some difference settings for audit policy and audit logs.

Audit policy

Path is a single YAML file: /etc/kubernetes/audit-policy.yaml
readOnly: API server only needs to read it
type: File: The file already exists, Kubelet should fail to start the apiserver if the file is missing

Audit logs

Path is a directory /var/log/kubernetes/audit/, API server writes out multiple log files (with rotation) audit.log, audit.log.1, etc
readOnly: false
type: DirectoryOrCreate: It may not exist yet on a fresh system, Kubelet will automatically create it so the API server can write logs

Policy

A Kubernetes audit policy has three important layers:

1. Resources you target

resources:
  - group: ""    # core API group
    resources: ["pods", "configmaps", "secrets", "persistentvolumeclaims", ...]

group: "" means Core API group, like pods, configmaps, secrets, services, pv, pvc.

but there are also non-core-api

Resource	Group
deployments	apps
daemonsets	apps
jobs	batch
ingresses	networking.k8s.io
certificatesigningrequests	certificates.k8s.io

so the resource eg.

resources:
  - group: "apps"
    resources: ["deployments"]

2. Which stages you log

stages: ["Request", "Response", "Metadata"]

Request: log the incoming request before it hits handlers
Response: log the full response (including responseBody if enabled)
Metadata: only logs the request metadata (verb, user, resource, namespace)

Normally not used, but we often define omitStages to reduce noise.

omitStages: ["RequestReceived"] is used to: Reduce extremely noisy logs with no loss of useful information.

Stage	Meaning
`RequestReceived`	kube-apiserver got the request
`ResponseStarted`	apiserver is sending stream response
`ResponseComplete`	finished sending full response
`Panic`	apiserver crashed during request

3. Log level fields

level: None | Metadata | Request | RequestResponse

3.1. level: None

No audit event produced, which is used to ignore noisy resources. eg.

- level: None
  resources:
    - group: ""
      resources: ["events"]

level: Metadata

Logs:

user (e.g., system:serviceaccount:default:sa)
verb (get, list, update, patch…)
resource (configmaps, pods…)
namespace
request URI
response code

No request/response bodies. This is lightweight and safe for Secrets.

3.2. level: Request

Logs everything from “Metadata” plus the request body. eg.

the new version of a ConfigMap
the PVC spec submitted by kubectl
the PATCH operations

Does NOT log the response body.

3.3. level: RequestResponse

Logs:

metadata
request body
response body

eg. when we run command kubectl apply -f my-pv.yaml

the audit logs include:

the PV YAML sent
the PV YAML that apiserver stored after admission

This is the most expensive but also the most complete.

3.4 Summary

Level	Logs	Notes
None	nothing	Useful to exclude noisy resources
Metadata	verb, user, resource, namespace, stage	Very cheap, no bodies
Request	request headers + request body	More expensive
RequestResponse	request + response body	Most expensive

CKS Notes - work node update

Cheedge Lee — Fri, 28 Nov 2025 17:21:19 +0000

Here just a quick present of basic workflow of how to upgrade on the work node.More details can refer to the official doc, upgrading Linux work node.

Notice: The official doc shows is the completed procedural which we will stop the jobs, and drain the node(s) then continue the upgrade, but here we do the upgrade one work node without disturbing the jobs.

1. check the nodes we will upgrade

controlplane:~$ k get no
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   10d   v1.34.2
node01         Ready    <none>          10d   v1.34.1

2. ssh to the node and check the version

controlplane:~$ ssh node01
node01:~$ apt-cache madison kubeadm
   kubeadm | 1.34.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages

3. ssh to work node, do the upgrade

node01:~$ apt install kubeadm=1.34.2-1.1
node01:~$ kubeadm upgrade node
node01:~$ apt install kubelet=1.34.2-1.1 kubectl=1.34.2-1.1
node01:~$ systemctl restart kubelet

4. back to controlplane and check the result

node01:~$ ssh controlplane
controlplane:~$ k get node
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   10d   v1.34.2
node01         Ready    <none>          10d   v1.34.2

Cluster upgrade

If need to upgrade the controlplane node, see the article here, or the official doc about upgrade kubeadm.

CKS Notes - TLS

Cheedge Lee — Thu, 27 Nov 2025 20:40:34 +0000

TLS

# generate TLS cert and key
openssl command [ options ... ] [ parameters ... ]

1. TLS config files

About TLS we will mainly focus on 3 levels/types encryption in this article.

inside cluster communication (/etc/kubernetes/pki/*)
client (kubectl) communicate with apiserver(~/.kube/config)
application level communication (TLS Secrets)

1.1 PKI files

PKI files in /etc/kubernetes/pki/ secure:

kubelet ↔ apiserver (mTLS)
scheduler ↔ apiserver
controller-manager ↔ apiserver
apiserver ↔ etcd
(in some setups) kube-proxy ↔ apiserver

These ensure control-plane communication is always encrypted and authenticated.

1.2 Kubectl using TLS

`~/.kube/config` stores:

the API server URL (HTTPS)
the CA certificate
user’s client certificate/key

This ensures:

kubectl connects to apiserver using TLS
apiserver knows which user is calling

1.3 Application level (TLS Secrets)

Stored inside Kubernetes as:

apiVersion: v1
kind: Secret
metadata:
  name: testsecret-tls
  namespace: default
data:
  tls.crt: base64 encoded cert
  tls.key: base64 encoded key
type: kubernetes.io/tls

Used by:

Ingress controllers (public HTTPS)
Gateways (Istio)

These certificates belong to applications, not Kubernetes.

1.3.1 TLS Secret and usage

kubectl create secret tls <NAME> --cert=/PATH/TP/<CERT> --key=/PATH/TO/<KEY>

and the Ingress use it

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-example-ingress
spec:
  tls:
  - hosts:
      - https-example.foo.com
    secretName: testsecret-tls
  rules:
    ...

and then we can use curl to verify

curl -kv https://...

more details check official doc

and more details about the secrets, can also check previous blog

2. TLS Flags

TLS flags apply to the server's accepted TLS standards, and the type of certificate (kubeconfig cert, PKI cert, bootstrap cert) doesn’t matter, as long as the TLS connection follows the rules.

ALL these must use:

allowed TLS version
allowed cipher suites
supported handshake parameters

2.1 TLS Flags of apiserver

apiserver is a TLS server and TLS flags define what ALL clients must follow.

Clients include:

kubectl → apiserver
kubelet → apiserver
controller-manager → apiserver
scheduler → apiserver
operators → apiserver

Example:

--tls-min-version=VersionTLS13
--tls-cipher-suites=TLS_AES_128_GCM_SHA256

2.2 TLS Flags of etcd

Example:

--cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

These apply to:

Clients connecting to etcd:

kube-apiserver
etcdctl (if used)
backup/restore tools
any etcd peer connections

Reference

The exact flags can refer to the k8s (kube-apiserver) and etcd (security) official docs.

CKS Notes - Image Security

Cheedge Lee — Thu, 27 Nov 2025 10:21:32 +0000

ImagePolicyWebhook

Like NodeRestriction, ImagePolicyWebhook is also a “Admission Controller” plugin.

As we said before, the “Admission Controller” works after authentication and authorization (RBAC).

Authentication → Authorization → Admission ( NodeRestriction/ImagePolicyWebhook)

NodeRestriction : Prevent nodes from modifying objects they are not allowed.
ImagePolicyWebhook : Validate container images used in Pods before the pod is admitted

Basically it checks before Kubernetes allows a Pod to run, and call an external image security service and ask: Is this image allowed?

1. External service

Here the external server works as:

Kubernetes API server performs an HTTP POST to an external webhook server (HTTPS endpoint) to ask:

“Is this image allowed?”

That server might be:

A company’s internal image scanner
A vulnerability scanner (Trivy server)
A signature validator (Notary v1/2, Cosign webhook)
A custom Python/Go program
A local service on the same node (like https://localhost:1234)
A remote service outside the cluster

The API server only sends image info.

The external service decides what to check.

2. Admission Controller Configure file

the admission controller configure file can be either in yaml/json

{
  "apiVersion": "apiserver.config.k8s.io/v1",
  "kind": "AdmissionConfiguration",
  "plugins": [
    {
      "name": "ImagePolicyWebhook",
      "configuration": {
        "imagePolicy": {
          "kubeConfigFile": "/path/to/<kubeconfig_file>",
          "allowTTL": 100,
          "denyTTL": 50,
          "retryBackoff": 500,
          "defaultAllow": false
        }
      }
    }
  ]
}

kubeConfigFile The kubeconfig file used by API server to call the external webhook
allowTTL Cache “allowed” decisions for 100 seconds
denyTTL Cache “denied” decisions for 50 seconds
retryBackoff If webhook fails, wait 500ms before retrying
defaultAllow = false If webhook cannot be reached → deny pods by default (strict mode)

3. KubeConfigFile

The kubeconfig tells the API server:

Which webhook server to call
Which TLS certs to use for mutual TLS
Which CA to trust for server verification

Example:

clusters:
- cluster:
    certificate-authority: /etc/kubernetes/policywebhook/external-cert.pem
    server: https://localhost:1234
  name: image-checker

users:
- name: api-server
  user:
    client-certificate: /etc/kubernetes/policywebhook/apiserver-client-cert.pem
    client-key:  /etc/kubernetes/policywebhook/apiserver-client-key.pem

Meaning:

API server must contact a webhook running on https://localhost:1234
API server will verify the webhook using the provided CA cert.
API server authenticates itself to the webhook using mutual TLS.

4. manifest kube-apiserver.yaml

/etc/kubernetes/manifest/kube-apiserver.yaml

apiVersion: v1
kind: Pod
metadata:
  ...
spec:
  containers:
  - command:
    - kube-apiserver
    - --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
    - --admission-control-config-file=/PATH/TO/<ADMISSION_CONTROLLER_CONFIG_FILE>
    # eg. /etc/kubernetes/policywebhook/admission_config.json
    ...

Reference

Official doc: Imagepolicywebhook

Also can check the examples here

Trivy Image scan

Before we talk about the Trivy can be the external server to check the images. Here we will see how to use Trivy to scan images.

# find the images
k get pod -oyaml | grep image:
k describe pod |grep -iE '^Name:|Image:'

# find the high,critical images
trivy image -s HIGH,CRITICAL <IMAGE_NAME>:<VERSION>
# we can also show the specified Vulnerability
trivy image -s HIGH,CRITICAL <IMAGE_NAME>:<VERSION> |grep <Vulnerability>
# eg.
trivy image nginx:1.19.1-alpine-perl | grep CVE-2021

Find the image with vulnerabilites, and we can either

remove the pod or scale down the deployment
patch the source of the image Dockerfile

# remove the pod or scale down the dp/sc
k -n <NAMESPACE> delete pod <NAME>
k -n <NAMESPACE> scale deploy <DEPLOY> --replicas 0

remove the pod or scale down the dp is just temporal way, in reality we still need to find a image which is no vulnerabilities to patch the source

# 1. find a base image
Report Summary

┌────────────────────────────────┬────────┬─────────────────┬─────────┐
│             Target             │  Type  │ Vulnerabilities │ Secrets │
├────────────────────────────────┼────────┼─────────────────┼─────────┤
│ morc-api:0.1-5 (alpine 3.22.1) │ alpine │       25        │    -    │
└────────────────────────────────┴────────┴─────────────────┴─────────┘

from the Report summary we can know it is based on alpine, so we can find the latest version for patch, but first check it.

$ trivy image nginx:alpine

and then build the image,

docker build -t <IMAGE>:<PATCHED_VERSION> -f <Dockerfile> .

And as Kubernetes does not use Docker anymore.(Kubernetes ≥ 1.24 does NOT run Docker.)

It uses:

containerd
or CRI-O

So even if we have built the image using docker build, that image exists only in Docker’s local storage, not in the cluster runtime.

This means:

docker build → Docker local image store

NOT visible to Kubernetes / containerd / CRI-O

Therefore we need to do following steps:

1. Save the Docker image

docker save <IMAGE_NAME>:<PATCHED_VERSION> -o <IMAGE_NAME>_<PATCHED_VERSION>.tar

2. Import into containerd

ctr -n k8s.io images import <IMAGE_NAME>_<PATCHED_VERSION>.tar

This loads the image into the runtime used by kubelet → containerd.

3. Check image in containerd

crictl images | grep <IMAGE_NAME>

If they do NOT import the image:

Kubelet won’t see the patched image
containerd will try to pull the old image from registry
Trivy scanning containerd won’t see the patched image

Reference:

Can check an example here

CKS Notes - some notes on docker(podman)

Cheedge Lee — Wed, 26 Nov 2025 16:22:08 +0000

1. Basic CMD

These basic cmd are same structure for both docker and podman.

# build image
$ docker build -t <NAME> -f /PATH/TO/Dockerfile <BUILD_CONTEXT>

$ docker image ls

# run docker container -d detached
$ docker run --name <NAME> -d <BASE_IMAGE>

# check running contianers
$ docker ps

# check process in container
$ docker exec <CONTAINER_NAME> ps
$ docker rm <CONTAINER_NAME> --force

<BUILD_CONTEXT>: the directory/path that Docker sends to the daemon, containing:

files referenced by COPY
files referenced by ADD
anything needed during build

In short, BUILD_CONTEXT is the dir/path when the Dockerfile try to do the “COPY” or “ADD” operations it will searched dir/path.

1.1 Tags

if you build the same image using docker build -t <SAME_NAME> -f /PATH/TO/Dockerfile <BUILD_CONTEXT>, it will build a new image, the old one’s tag will be removed as <none>.

$ docker image ls
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
base-image   latest    2487f22f50cb   5 seconds ago    5.58MB
<none>       <none>    e9a4699a5cbb   17 minutes ago   5.58MB
<none>       <none>    8a8330c3730a   23 minutes ago   5.58MB

1.2 Layers (Caching)

More layers → more caching → faster builds.

Docker creates a new layer for:

FROM
RUN
COPY
ADD

BUT NOT for:

CMD
ENTRYPOINT
ENV
EXPOSE
USER
WORKDIR

Notice: CMD and ENTRYPOINT DO NOT create layers. They are metadata, not filesystem layers.

Example:

RUN apk update
RUN apk add curl
RUN apk add git

Each of these caches independently.

If merging them together:

RUN apk update && apk add curl git

Then any small change breaks the entire cache.

So sometimes more layers = faster builds.

Notice: I don’t mean more layers are always good, more layers will create more temporal artifacts. So reduce layers where possible, but caching and maintainability matter more than minimizing layers.

1.3 share PID


docker run --name sidecar --pid=container:app -d busybox sleep infinity

apiVersion: v1
kind: Pod
metadata:
  name: example
spec:
  shareProcessNamespace: true
  containers:
    - name: app
      image: nginx
    - name: sidecar
      image: busybox
      command: ["sleep", "3600"]

2. Dockerfile Best Practice - Multi-stage build

# syntax=docker/dockerfile:1
FROM golang:1.24
WORKDIR /src
COPY <<EOF ./main.go
package main

import "fmt"

func main() {
  fmt.Println("hello, world")
}
EOF
RUN go build -o /bin/hello ./main.go

FROM scratch
COPY --from=0 /bin/hello /bin/hello
CMD ["/bin/hello"]

the second FROM scratch will build on first FROM golang:1.24, as “The COPY --from=0 line copies just the built artifact from the previous stage into this new stage.”

more details check here.

3. Dockerfile Best Practice - Run as non-root user

RUN addgroup -S appgroup && \
    adduser -S appuser -G appgroup -h /home/appuser
# or for Alphine
RUN adduser -D -g '' appuser

Both are secure as long as you do USER appuser.

The explicit version gives more control, but the simpler version is secure enough for typical apps.

-D : create user with no password, no shell, minimal config
-g '' : empty GECOS field

4. Dockerfile Best Practice - Never echo secrets in Dockerfile

example

Layer 1: RUN echo $TOKEN > /tmp/token
Layer 2: RUN sh -c 'register.sh /tmp/token'
Layer 3: RUN rm /tmp/token

each RUN will create a new layer, so if we use cmd to save “TOKEN”, which means it will be saved inside that layer, even then using RUN rm command. Therefore, DO NOT save secrets in Dockerfile.

# syntax=docker/dockerfile:1.2
RUN --mount=type=secret,id=token \
    sh -c 'cat /run/secrets/token | register.sh'
# or
RUN register.sh $TOKEN

For better practice, secret can be passed into the container during runtime as env variable TOKEN

5. Docker related Security best practice

basic configuration files:

/usr/lib/systemd/system/docker.socket
/usr/lib/systemd/system/docker.service

if you forgot the path, just check their status, eg. systemctl status docker.socket.

After changing the setting, remember to restart corresponding service.

systemctl daemon-reload 
systemctl restart docker.socket 
systemctl restart docker.service

5.1 Do NOT expose Docker over TCP

Otherwise it will expose ROOT ACCESS without authentication.

Edit /usr/lib/systemd/system/docker.service, remove -H tcp://0.0.0.0:2375 from ExecStart.

# remove -H tcp://0.0.0.0:2375

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

check the TCP

ss -tlnp | grep dockerd

5.2 Enforce TLS if you must expose TCP

If remote Docker API is absolutely required:

enable TLS
require certificates
never use TCP without TLS

Example:

-H tcp://127.0.0.1:2376
--tlsverify
--tlscacert=ca.pem
--tlscert=server-cert.pem
--tlskey=server-key.pem

5.3 Run dockerd as root (default), but avoid giving docker group to users

docker group = root equivalent

in /usr/lib/systemd/system/docker.socket set the

[Unit]
Description=Docker Socket for the API

[Socket]
ListenStream=/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
# either set SocketGroup=root or docker 

[Install]
WantedBy=sockets.target

Never add regular users to docker:

# check user belonging group
group <USER_NAME> # eg. group docker

# show all groups
getent group
vi /etc/group

# print real and effective user and group IDs
id <USER>
vi /etc/passwd
# $ id ubuntu
# uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),24(cdrom),27(sudo),30(dip),105(lxd)

# delete teh user from group
gpasswd -d <USER> <FROM_GROUP>
gpasswd -d <USER> docker
groupdel docker
# then socket file created with group = root

5.4 Use AppArmor or SELinux

Hardens container isolation.

Ubuntu:

# --security-opt apparmor=default
docker run --security-opt apparmor=myprofile ...

RHEL-based:

# --security-opt label=type:container_t
docker run --security-opt label:type:svirt_lxc_net_t ...

which is similar as Kubernetes-level AppArmor

In Kubernetes, we do NOT call docker run. We must annotate the Pod:

metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/mycontainer: localhost/myprofile

5.5 Restrict container capabilities

Capabilities = parts of root’s power.

Remove everything, then add only needed:

docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE ...

which is similar with Kubernetes-level capabilities

In Pods:

securityContext:
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]

Kubernetes simply passes these to the container runtime (Docker, containerd, CRI-O).

5.6 Make `/var/lib/docker` read-only

Prevents tampering.

--read-only

5.7 Enable logging limits

Avoid log spamming or disk filling.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

CKS Notes - Pod Security

Cheedge Lee — Tue, 25 Nov 2025 19:29:17 +0000

For Pod Security, there are basically two layers/types:

Pod security settings for each pod
namespace labels for Pod Security Admission (PSA)

Layer A: Pod security settings

Defined in:

spec.containers[*].securityContext
spec.securityContext

This is container-specific + pod-wide security.

It sets the runtime security for a container.

Examples:

runAsNonRoot
runAsUser
allowPrivilegeEscalation
readOnlyRootFilesystem
capabilities
seccompProfile
privileged
fsGroup

Layer B — Namespace-level Pod Security Admission (PSA)

Defined using labels:

pod-security.kubernetes.io/enforce: restricted|baseline|privileged

This sets global rules for all pods in the namespace before creation.

restricted : strongest security, disallow most risky features
baseline : medium security
privileged : allow almost anything

work together

PSA (namespace label) is the policy gate
Pod securityContext is the actual behavior inside the pod

Even if the pod sets something insecure, PSA can block it.

Best practice harden rules

Here are the some pod hardening rules for best practice:

0. Namespace

restricted

1. Run as non-root

runAsNonRoot: true
runAsUser: 1000

Avoid privileged UID 0.

2. Disallow privilege escalation

allowPrivilegeEscalation: false

Stops processes from gaining higher privileges (e.g., via setuid binaries).

3. Drop unnecessary Linux capabilities

capabilities:
  drop: ["ALL"]

Add only what you need.

4. Use a read-only root filesystem

readOnlyRootFilesystem: true

Prevents writes to /, reduces persistence and attacks.

5. Use seccomp (system call filtering)

seccompProfile:
  type: RuntimeDefault

Prevents dangerous syscalls.

6. Avoid privileged containers

privileged: false

Never run privileged unless absolutely needed.

7. Use AppArmor (if available)

annotations:
  container.apparmor.security.beta.kubernetes.io/nginx: runtime/default

8. Restrict host access

Avoid these unless needed:

hostNetwork
hostPID
hostIPC
hostPath volumes

Reference

Official doc:

CKS Notes - ServiceAccount

Cheedge Lee — Mon, 24 Nov 2025 19:46:03 +0000

Previous Notes about the Apiserver request security we have shown two identities (node identity /etc/kubernetes/kubelet.conf and admin cluster identity ~/.kube/config ). Now we will talk about another identity, it is used for the Pods — ServiceAccount. Actually we have talked about ServiceAccount before in previous post about RBAC(for more details about RBAC pls check here).

RBAC

RBAC is an authorization process, which will grant the Service Account with permissions (Role, ClusterRole).

SA - RoleBinding - Role → allowed actions

ServiceAccount: identity
Role or ClusterRole : ns-scoped or cluster-scoped allowed operations (permissions)
RoleBinding, ClusterRoleBinding

quick recap:

k create clusterrole NAME --verb OPS1,OPS2 --resource RESOURCENAME1,RESOURCENAME2
k create rolebinding NAME --role ROLENAME --serviceaccount SANAME
# do not miss up the --as and --service (--as is kubectl param used for test)
k auth can-i VERB RESOURCE --as system:serviceaccount:NAMESPACE:NAME

Pod → Apiserver

Any pod can connect to the API server, but without RBAC permissions it cannot do anything meaningful.

1. Pod (no API server access)

Do not bind SA to any role (default sa)
Ideally disable token mounting
Pod runs with minimal risk

2. Pod → API Server

Create custom ServiceAccount
Bind it to Role/ClusterRole
Assign that SA to the pod(s)

For case 1, we need to opt out the of API credential auto-mounting

either set on sa

apiVersion: v1
kind: ServiceAccount
metadata:
  name: build-robot
automountServiceAccountToken: false
...

or set on the pod needed

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: build-robot
  automountServiceAccountToken: false
  ...

more details pls refer to the official docs “opt out of API credential auto mounting”.

Verify mount

# if not mount, will show nothing
kubectl -n test exec -it podA -- mount | grep serviceaccount
# otherwise, if it still mount the token will show sth like:
tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime,size=2097536k,inode64,noswap)

kubectl -n test exec -it podA -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
# cat: /var/run/secrets/kubernetes.io/serviceaccount/token: No such file or directory

Apiserver ←→ Kubelet (Pod)

API server communicates with pods indirectly via kubelet and kubelet has its own identity (client certificate from kubelet.conf)

Examples:

API server tells kubelet to create a pod
API server watches pod status
API server updates pod info
Kubelet sends logs to API server
Kubelet mounts volumes/secrets
Kubelet reports liveness/readiness
API server fetches metrics from kubelet

Identities comparision

1. Human / Automation (kubectl, CI)

Identity comes from kubeconfig, using:

client cert
or token (OIDC, bearer)

2. Kubelet (Node agent)

Identity comes from:

/etc/kubernetes/kubelet.conf
client certificate
User identity: system:node:<nodeName>
Group: system:nodes

3. Pod (Workload)

Identity comes from:

ServiceAccount token projected by kubelet

Identity becomes: system:serviceaccount:<ns>:<name>

Workflow

SA token is a JWT token, so basically we can understand this process as a JWT token verify process.

1. send request

k get pod -n test

then kubectl actually send the request

GET /api/v1/namespace/test/pods
Authorization: Bearer <JWT_TOKEN>

That token is a JWT with 3 parts:

Header – algorithm & token type
Payload (claims) – SA identity information
Signature – signed with API server private key

Payload includes:

iss → issuer (kubernetes.default.svc)
sub → service account identity (system:serviceaccount:<namespace>:<sa-name>)
kubernetes.io/serviceaccount/namespace
kubernetes.io/serviceaccount/uid
Expiry, audience, etc.

2. Apiserver handle

2.1 Authentication

look for key to verify the SA JWT token, if it is authenticated, go next

2.2 Authorisation

RBAC authorization

request verb = get
resource = pods
namespace = test
user = system:serviceaccount:test:<sa-name>

2.3 Admission Control (NodeRestriction)

2.4 Do the Operation

Summary

ServiceAccount:

Pod Identity (JWT token)
SA - RoleBinding - Role → allowed actions
automountServiceAccountToken
Pod → Apiserver
Apiserver verify token
verify kubectl -n test exec -it podA -- mount | grep serviceaccount

CKS Notes - Apiserver request security

Cheedge Lee — Sun, 23 Nov 2025 21:17:54 +0000

1. Identity = kubeconfig

Here the identity equals kubeconfig file

Every kubeconfig file specifies:

the certificate
the client key
the username (CN)
the group (O)
the cluster endpoint

This determines:

Who you are (identity)
What you can do (RBAC permissions)

Now let’s first distinguish 2 types of identity: admin cluster identity vs node identity.

1.1 kubelet.conf

Location:

each nodes /etc/kubernetes/kubelet.conf

Identity inside it:

user = system:node:<nodeName>
group = system:nodes

Meaning:

This identity belongs to the kubelet on that node

Permissions (through RBAC):

can update its own node object
can update pod status for pods assigned to it
cannot get/delete/list arbitrary pods
cannot label other nodes

This identity is heavily restricted by NodeRestriction.

1.2 `$HOME/.kube/config` (`admin.conf` copy)

Location:

on control plane node

Identity:

user = kubernetes-admin
group = system:masters

Meaning:

This is the cluster admin identity
Full power (cluster-admin cluster role)

Permissions:

can do “everything” — (get/delete/list/create/patch ANY resource)

NodeRestriction does NOT apply to this identity.

2. Workflows

there is one section about Admission control phase, but it may not easy to understand. Here is my previous post for the workflow (Kubernetes workflow).

2.1 Basic concepts

Let’s first have a overlook of some basic components.

kubelet

runs the containers
updates pod/node status

kubectl

send requests

kubelet.conf = identity of kubelet

Used by kubelet when it talks to API server, (NodeRestriction)

admin.conf / ~/.kube/config = identity of cluster admin

Used by kubectl

2.2 Diff workflows

Now let’s see the workflow for diff process

API request:

kubectl → API server → etcd

Pod scheduling:

scheduler → API server → etcd

Pod execution:

kubelet → containerd → pod runs

Status updates:

kubelet → API server → etcd

Controllers maintain desired state:

controller manager → API server → etcd

3. Security

the basic security flow is Authentication - Authorization - (Admission Control)

3.1 Authentication (Identity)

Here we see when using these 2 type (node identity, cluster identity) config file their workflow:

When human/admin runs kubectl (on controlplane)

# eg. on controlplane
k get node
k label node node01 test/thislabel=123

0. ~/.kube/config

1. kubectl sends HTTPS request with admin cert

2. API server authenticates admin identity

When kubelet talks to API server (on each node)

# eg. on node01
k label node node01 test/thislabel=123 --kubeconfig=/etc/kubernetes/kubelet.conf

0. /etc/kubernetes/kubelet.conf  (local to that node)

1. kubelet loads cert/key

2. kubelet sends HTTPS request to API server

3. API server authenticates "system:node:NODENAME"

notice:

kubectl sends requests for user operations

kubelet sends requests for node/pod management

3.2 Authorization (RBAC)

Uses CN and O from the certificate
Checks “is this identity allowed to do this verb on this resource?”

RBAC uses this identity to check:

Which User? eg. system:node:node01
Which Groups? eg. system:nodes (all kubelets), system:authenticated
Which verb/resource?
- update node/worker1
- patch node/worker1
- get pods
RBAC rules that apply to those groups
- eg. Kubernetes has a built-in ClusterRole

more details can refer to official doc or previous post here.

3.3 Admission Control (NodeRestriction)

Check the data/operations that try to modify the resource in the arriving request. Prevents kubelets from doing dangerous modifications:

Node labels
Node annotations
Node taints
Pods on other nodes

More details can check in official doc “Admission Control in Kubernetes” and “Using Node Authorization”

CKS Notes -- Kube-bench

Cheedge Lee — Fri, 21 Nov 2025 10:02:03 +0000

With the experience shared by people on the internet, I summarized some key aspects as a series of articles for preparing the CKS exam. Each aspects should be in short concise points, so this is not a detailed tutorial, just some practical reminders.

Notice: some concepts are based on my understanding, it may be not accurate or even correct, therefore this is just a handbook when I was preparing the CKS exam.

kube-bench is a tool to check if the k8s cluster fulfilled the CIS security benchmark.

Concepts

Basic Command:

ssh NODE
kube-bench run --targets TARGETS --check VERSION

params:

1. check `--targets`:

master
node
controlplane
etcd
policies

2. check CIS version `--check`

Checking items

1. on `master`

kube-bench run --targets master

Apiserver (/etc/kubernetes/manifests/kube-apiserver.yaml)
ControllerManager (/etc/kubernetes/manifests/kube-controller-manager.yaml)
PKI directory (/etc/kubernetes/pki/)
Schedualer (/etc/kubernetes/manifests/kube-scheduler.yaml)

2. on `node`

ssh NODE
kube-bench run --targets node

`kubelet` is considering as node-level component

it mainly checks kubelet related configs:

/var/lib/kubelet/config.yaml
/etc/kubernetes/kubelet.conf
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
kubelet certificate location
anonymous auth
webhook authz
protecting /var/lib/kubelet/
TLS bootstrapping config
client CA
permissions (644/600)

Notice: need manually restart kubelet

3. `etcd` check

we only focus on kubeadm cluster ( for cloud, they will not expose etcd, and for external managed etcd cluster, ssh to the node)

kubeadm will assign the etcd to the controlplane node.

ssh CONTROLPLANE_NODE
kube-bench run --targets etcd

/etc/kubernetes/manifests/etcd.yaml

Authentication enabled
1. --client-cert-auth=true
2. --peer-client-cert-auth=true
Encryption enabled
1. --cert-file
2. --key-file
3. --peer-cert-file
4. --peer-key-file
Proper paths
1. /etc/kubernetes/pki/etcd/

notice: for kubeadm cluster, kubeadm will update the /mainfests and then kubelet will auto restart etcd, there is no need to manually restart it.

Notice:

here we should notice command: kube-bench run --targets node , for kube-bench run --targets master or other targets:

master : API server, controller, etc —kubelet watches the manifest files
etcd : etcd services — kubelet watches the manifest files
policy: kubectl

the kube-apiserver, kube-controller-manager, kube-scheduler, etcd under kubeadm cluster will managed by kubeadm/kubelet , the config file are under /etc/kubernetes/manifests/*

And the policy is control by kubectl, so these we can just follow the recommendations which kube-bench shows.

Component	How it runs	Config change effect	Restart needed?
kube-apiserver	Static pod	Kubelet watches manifest	No (auto restart)
kube-controller-manager	Static pod	Same	No
kube-scheduler	Static pod	Same	No
etcd (kubeadm)	Static pod	Same	No
policies	YAML API objects	Apply with kubectl	No restart
kubelet	systemd service	Reads config only at startup	Yes — manual restart

while for kubelet related configs we need to find the kubelet config file first, and then find the environment file location for fixing.

# find kubelet config file
systemctl status kubelet
# find the env para settings file location
# eg. the kubelet config is: /var/lib/kubelet/config.yaml, then inside it:
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# change the params in the corresponding file.

Do not directly fix the params in the Environment: .

Summary

Benchmark section	Contains checks for	kube-bench target
Master Node (1.x)	API server, controller-manager, PKI, scheduler, etc.	`master` or `controlplane`
Node (4.x)	Kubelet, kubelet config, certificates, permissions	`node`
etcd (3.x)	etcd service, certs, ports, flags	`etcd`
Policies (5.x)	PodSecurityPolicies (old), security policies	`policies`

From one CSRF case to see handling third-party cookie blocking in browser

Cheedge Lee — Thu, 16 Oct 2025 17:22:44 +0000

Recently I am migrate my previous website for “Lieben in Deutschland” test frontend to another cloud. And build for my new backend demos website. But during these, I met with some issue with CSRF, and then I found this interesting topic of “third-party cookie blocking”. By this blog I want to record some steps of solving these issue.

1. CSRF

Let’s first have a quick review of CSRF attack and its solution.

1.1 CSRF explaination

We use CSRF as the case, which is the “Cross Site Request Forgery”, where the attacker can utilize the login user’s authenticated browser stored cookies to forge/trick a request and send to the server.

1.2.1 X-XSRF-Token

As the cookie can be utilized to authenticate the malicious form submission, so we can add a token in both the cookie and the Headers.

Even the cookies token stored cannot be check by the attacker, but it can still be used to send to the server, so what we actually need, is checking the token inside the Header with the session token on the server.

The Session Link Always Works: The Session Cookie sent by the browser always matches the server's session store (assuming the user is logged in). This cookie is what links the incoming request to the victim's authenticated session on the server.
The Token Mismatch is the Defense: The server checks if the token inside the request body/header matches the token stored within the server-side session data (which the Session Cookie points to).

1.2.2 .Net solution

in .Net we use Antiforgery, where we can add the Header name as:

builder.Services.AddAntiforgery(options =>
{
    options.HeaderName = "X-XSRF-TOKEN";
});

and for MVC we can directly use the middleware ValidateAntiForgeryToken, and for web api, we can use the IAntiforgery for the validation.


    [ApiController]
    [Route("api/[controller]")]
    public class CsrfDemoController : ControllerBase
    {
        private readonly IAntiforgery _antiforgery;

        public CsrfDemoController(IAntiforgery antiforgery)
        {
            _antiforgery = antiforgery;
        }

        [HttpGet("token")]
        public IActionResult GetCsrfToken()
        {
            var tokens = _antiforgery.GetAndStoreTokens(HttpContext);
            return Ok(new { csrfToken = tokens.RequestToken });
        }

        [HttpPost("secure-action")]
        public async Task<IActionResult> SecureAction([FromBody] dynamic data)
        {
            // diff from MVC: Manual validation here, not using
            // [ValidateAntiForgeryToken]
            await _antiforgery.ValidateRequestAsync(HttpContext);

            return Ok(new { message = "CSRF token validated!", input = data });
        }
    }

ASP.NET Core's AddAntiforgery validation requires BOTH:

A cookie with a value
A header/form field with a matching value

It validates that they match.

1.2.3 JWT, etc

Here we just one line notice that actually if we use JWT or other more secure way for login authentication, the CSRF will be avoid (even though the third party cookie as what we described in this article, as long as it not stored in third party cookie). But this is not our topic here.

2. Occurence

How this related to the third party cookie blocking issue of browsers, for this I need to move to my recently re-deployment. In order to give a clear workflow, let’s focus on the main services, (So here we don’t mention API Gateway or other secure settings, also let DB alone). Use a simple and clean structure.

2.1 Basic Architecture

My new deployment is:

Backend: AWS Lambda (.Net 8)
Frontend: Netlify (React)

the backend code basically is like above .Net code, the frontend just some fetch calls just like:

// GET
await fetch(`${apiUrl}/api/CsrfDemo/token`, {
    method: 'GET',
    credentials: 'include', // very important for cookie
});
// POST
await fetch(`${apiUrl}/api/CsrfDemo/secure-action`, {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json',
        'X-XSRF-TOKEN': csrfToken, // antiforgery header, set in .net code
    },
    credentials: 'include', // sends the cookie
    body: JSON.stringify({ message: 'CSRF test from React' }),
});

2.2 What happening?

Before the deployment, I tested all the APIs locally, all APIs work well, CSRF API either.

But after the deployment, I found CSRF API response in the browser will show me server error like:

{
    "type": "https://tools.ietf.org/html/rfc9110#section-15.6.1",
    "title": "An unexpected error occurred",
    "status": 500,
    "detail": "The required antiforgery cookie \".AspNetCore.Antiforgery\" is not present.",
    "traceId": "00-53a3ef57a071175d1c7dd256494f9d2c-1af0e8de357b09p7-00"
}

it told me .AspNetCore.Antiforgery is not shown in my cookie.

That’s wired, because locally it test well, and even after deployment, when I use curl to call CSRF API, it also works well and show .AspNetCore.Antiforgery in cookies.

# 1. Get token and save cookies
curl -v -c cookies.txt \
  -H "Origin: https://myfrontend-addr" \
  https://my-lambda-url/api/CsrfDemo/token

# Check if cookies.txt has the antiforgery cookie
cat cookies.txt

# 2. Use the saved cookies
curl -v -b cookies.txt \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-XSRF-TOKEN: <token-from-above>" \
  -H "Origin: https://myfrontend-addr" \
  -d '{"data": "test"}' \
  https://my-lambda-url/api/CsrfDemo/secure-action

and the cookies.txt file show me:

# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

#HttpOnly_mylambda.lambda-url.on.aws    FALSE   /   TRUE    0   .AspNetCore.Antiforgery abcdefg1234567890

it shows clearly the .AspNetCore.Antiforgery, which include the CSRF token eg. “abcdefg1234567890”.

So for now, as locally test successfully, my backend code logic is fine, and since curl works, this confirms my Lambda backend is configured correctly. The issue maybe browser-specific, browsers have stricter cookie policies than curl, especially for cross-origin requests.

2.3 Third Party cookies blocking (not store)

Let’s see the browser request and response.

By checking the Network, we found the for the GET token API response shows:

HTTP/1.1 200 OK
Date: Wed, 15 Oct 2025 16:42:20 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 171
Connection: keep-alive
x-amzn-RequestId: 1234567890
access-control-allow-origin: https://my-frontend-url
x-frame-options: SAMEORIGIN
Set-Cookie: .AspNetCore.Antiforgery=abcdefg1234567890; path=/; secure; samesite=none; httponly
cache-control: no-cache, no-store
X-Amzn-Trace-Id: Root=abcd;Sampled=0;Lineage=1:80aab7f3:0
pragma: no-cache
access-control-allow-credentials: true

where we see Set-Cookie Header and the .AspNetCore.Antiforgery with CSRF token, eg. “abcdefg1234567890”.

while as previous React code we also see we already used:

credentials: 'include',

which should include the cookie, and browser then stored the cookie.

But when we check the cookies in the browser, we found empty, which means the browser hasn’t store the cookie.

This is different from when we check the local deployment case, in local testing, after the get token, we will found the cookies are stored with the name .AspNetCore.Antiforgery , which is same as what we see in above curl cmd.

So here comes the third Party cookies blocking.

Actually the name third party cookies blocking actually is the browser will not store the cookies from third party.

Comparing with previous flow, now it looks like:

There is no cookie stored as it is third-party cookie, consequently, no cookie will be sent.

3. A simple Solution

Now our situation is:

GET request (works):

Browser sends GET to Lambda
Lambda responds with Set-Cookie: .AspNetCore.Antiforgery=...
Browser receives it BUT doesn't store it (third-party cookie blocking)
However, the response body still contains the token as JSON

POST request (fails):

Browser has no cookie stored (because it was blocked)
Browser sends POST with X-XSRF-TOKEN header
Lambda checks: "Where's the cookie?" → Not found → Error

The Set-Cookie appears in response headers, but the browser silently refuses to store it.

The reason caused this is about our .Net code, as we point out previous that ASP.NET Core's AddAntiforgery validation requires BOTH:

A cookie with a value
A header/form field with a matching value

It validates that they match.

But since the cookie isn't stored, validation will fail.

3.1 Token comparison only from Headers

As the AddAntiforgery validate fails as the cookie are not stored, but we still want to validate the token, why not change the logic to only check the tokens from header manually, as the token in cookies will always same.

So we extract the token from Header X-XSRF-TOKEN and compare with the backend stored token.

Therefore we have the method re-write:

using Microsoft.AspNetCore.Mvc;

namespace backend.CSRF.Controllers
{
    [ApiController]
    [Route("api/[controller]")]
    public class CsrfDemoController : ControllerBase
    {
        // In-memory token storage, if focus on concurrency can use Redis to store
        private static readonly Dictionary<string, DateTime> _tokens = new();
        private static readonly object _lock = new();

        [HttpGet("token")]
        public IActionResult GetCsrfToken()
        {
            // Generate random token by ourselves manully
            var token = Guid.NewGuid().ToString() + Guid.NewGuid().ToString();

            lock (_lock)
            {
                _tokens[token] = DateTime.UtcNow.AddMinutes(10);
            }

            return Ok(new { csrfToken = token });
        }

        [HttpPost("secure-action")]
        public IActionResult SecureAction([FromBody] dynamic data)
        {
            // Get token from header manully
            var token = Request.Headers["X-XSRF-TOKEN"].FirstOrDefault();

            if (string.IsNullOrEmpty(token))
                return BadRequest(new { error = "Missing CSRF token" });

            lock (_lock)
            {
                // Check if token exists and is not expired
                if (!_tokens.TryGetValue(token, out var expiresAt))
                    return BadRequest(new { error = "Invalid or already-used CSRF token" });

                if (DateTime.UtcNow > expiresAt)
                {
                    _tokens.Remove(token);
                    return BadRequest(new { error = "CSRF token expired" });
                }

                // Remove token (single-use)
                _tokens.Remove(token);
            }

            return Ok(new { message = "CSRF token validated!", input = data });
        }
    }
}

now the token from GET method will be checked on the Network, Preview Tab, and in the POST method Request Headers, we can also find the X-XSRF-TOKEN header.

Summary

In this article we use a simplified CSRF case to show this third party cookie blocking in Browser issue, and how to solve this problem via a simplified way. Actually, there are many more secure and better way to solve this issue, there are many article talk about it, eg. Microsoft Learn (How to handle third-party cookie blocking in browsers). For this article just a record of what issue I have met with during the re-deployment.

A simple but practical demo of Web Api in C#

Cheedge Lee — Sun, 12 Oct 2025 20:03:23 +0000

only show the basic steps, as for the other part, version control, CICD, Unit Test, just let it alone, we will focus on the basic structure of the project and the optimise of our code.

initial project
architecture design CQRS, DDD, ES
EF and DTO mapping
- DB design, Code First
DIP & IoC
CRUD implement
Extra settings: log, api document

Again, I should say it is just a simple demo for newbies to come familiar with these basic concepts, and it’s just some basics from a junior devs daily work. Therefore it is also suitable for someone who want to find a junior dev’s position. If you want more detailed info about each aspect, and deeper philosophy behind the scene, I really cannot help, I am just the screw of the machine, not the driver...

1 Initial Project

By using Visual Studio to create an ASP.Net project, we can directly using its UI, select WebApi project, here we choose to use full web api instead of mini one. More conveniently, we can use command line dotnet new webapi with name to create the project.

2 Architecture Design

2.1 CQRS

The relative more modern architecture to build the WebApi project is CQRS (Command Query Responsibility Segregation), which you can just analogy it with the Read/Write separation in DB management, where command is the write operations (Create, Update, Delete), while query is stand for the read operation, therefore combining the write (create, update, delete) and the read query(retrieve) we can get all the operations of data.

2.2 DDD

DDD stands for Domain Driven Design, in more human-readable words, making the object readable to real world business.

DDD is a separated layer, which should deal with the basic business logic, instead of handling the entity, DTOs mapping.

On architecture aspect, we will see later, when we talk the relationship between about EF entity and DTO class. DDD domain model is an extra layer between EF entity and DTO class.

Entity <--> Domain layer <--> DTO

Repository/Reader/Writer -> Entity and Domain

Handlers -> Domain and DTOs

2.3 ES

Normally we keep a table to record the final results of the entity, for example, for each account we will have a current balance, this is the most common way we see, the table have only the final status of the entity. And this is a good way for read data, where we can directly get the current state. While on the other hand, a single final state which means squash all the previous states, when we try to roll back, there will be problems [1], to fix this, here comes Event-Sourcing, ES is a approach to record each operations so that the states can be build depend on each steps rather than the only final states.

[1] Notice: here we only want to explain the necessary of ES, for roll back or recovery, there are many other approaches without using ES. For example, for a traditional financial system, we always have the extra journal/ledger/audit tables in backoffice systems, which makes the system more robust, and secure. There are many materials, so we don’t expand it here, also I will have another note to discuss about it.

Here in order to further reduce the coupling, we will separate the event and the aggregate, therefore, we will have a separated EventContext and the other Entity DbContext.

2.4 Architecture choice

Feature specific

DDD type

3 EF and DTO

The basic functionality for a web api is to make the decoupling of the frontend and backend, and provide some standard entry for frontend to use to CRUD the data. And this decoupling approach we will see everywhere, and in some extent, all SOLID principles are related with decoupling more or less, in my view. Later we will see the DIP, which is the significant loose coupling approach. Here by using EF entity and the DTO class, we could realise that not mixing up the properties which the DB owns with the properties that can be directly used in CRUD, so that we will not expose all the properties, only provide the necessary entry.

3.1 Entity Framework

For EF entity, we could using the well established Entity Framework (choose the compatible version with your .net version) for this data access process. Here we met with another choices: Code First Design vs DB First Design.

Code First Design vs DB First Design

Here I don’t want to explain too much on these two concepts, if you like, there are many documents, blogs, tutorials discuss the difference and details about the two methods. We just list some common way to choose:

DB already exist, or you prefer to write SQL use DB first.
for new project, Code First is a good choice.

we can choose the Code First approach, and we could either use

Attribute Annoation
Fluent Api (edit in OnModelCreating method)

for the table creating. And then using the dotnet ef migration command for the initial, add, update, remove. It’s very convenient for users to manage the database.

3.2 DTO

After automatically generating the entity, as we discuss above, it’s better to use another layer for apis to use the entity, which is the DTO. We can do all the operations on the DTO, and the DTO should include only the necessary properties which we want to operate on. Moreover, we also need to build the bridge between entity and the DTO class, and we could also build a separated Mapper class by using the automapper or you can crated your own automapper via reflection/expression tree, (if you are interesting in this, pls check my another blog [here]).

Read/Write DTO

One step further, if we use the traditional CRUD methods, we could also separate the DTO into Read/Write DTOs, which means when we do the Get method, we can use the Read DTO, where only stores the properties Get method needs; while when we use Create/Update/Delete methods, we will change the data, some of the immutable properties should not be include, therefore we will choose the Write DTO.

3.3 Domain Model

As we discussed in DDD section, a concrete example for DDD implementation is the Domain Model layer between EF Entity and the DTO. Rather than including simple properties, Domain Model are more linking with business entity, aggregate, logic, further more, it also has the behaviours.

For example, the

Instead of directly building the mapper between entity and DTO, we break it into two steps:

EF Entity <--> Domain Model

Domain Model <--> DTO

For each steps build the mapper and then we can still use the DTO for all operations.

If take it more rigorous, we should say there is another mapping,

DbContext <--> EF Entity

which we already discussed in above section Code First Design vs DB First Design.

Notice: here we only separate the Entity which will be exposed through a public API or send them over the wire, which means the events entity should be internal, as we don’t want those events to expose as a public API. These internal-only events (for projections, logging, integration handlers) don’t need DTOs, so they work directly with our IDomainEvent types and the EventEntity persistence model.

4 DIP & IoC (DI)

4.1 DIP

As we discussing in above sections, in order to loose coupling so that making code more flexible for changing and extension. Additionally, it also easy for Unit Test. Still, there are many books, tutorials, blogs to explain the necessary, advantage, and the details of DIP, so we will not discuss it deeply here ([here] is also an unfinished tutorial for some basic info). What I want to simply show is the two basic approaches for implementations in web api.

4.1.1 Architecture Level

On architecture level decoupling, we will extract the corresponding interface of each layer, for example, for the Business Logic Layer(BLL) we can extract a separated interface layer for it (as IBLL), so that we can reduce the coupling.

4.1.2 Component (class) Level

On component level decoupling, instead of directly using the instance of the class, we will use the interface during the declaration.

4.2 IoC (DI)

And we will also discuss another related concept IoC and DI, here DI stands for dependency injection. More accurately, DI is one concrete form of IoC. As the name suggests, IoC is to give the control to the other — the invoke side.

4.2.1 Architecture Level

On Architecture level, IoC mainly reflects on the usage of container. When we initial the web api, we could found the built-in container in the main entry (Program.cs). Individually, we can also use other containers, most common containers as: Castle.Windsor, Unity, Autofac.

4.2.2 Component (class) Level

On component (class) level, we could see the introducing of interface in method parameters. Instead of directly using the class as the parameter, we will use the interface during the parameters passing. By this, we will give the control to the invoking side.

5 CRUD

Till now all the entity, model, dto has been created, then we can build the reader/writer/handlers

5.1 Pure Reader/Writer

Here by "pure" I mean these readers/writers are purely responsible only for communicating with database, so the operations here are clear CRUD ops. And the logic for readers/writers should be complete and simple for basic ops with DB.

Readers Methods

GetEntityById
GetEntitiesByField
GetEntitiesByMultiFields
GetAll

Writers Methods

AddNewEntity
UpdateEntity
DeleteEntity
SaveChanges

By this, we can not only make our Unit Tests easier, but also separate the Business Logic and the Data Access more clearly, which means we will put all business logics into the Query/Command Handlers and only need to do unit test on the handlers.

5.2 Repository ( DDD+ES )

By using DDD with ES, we normally not directly deal with the CRUD operations, instead we will accumulate the events to get the final status. Therefore we will put the read and write operations into one repository, and there are two basic ops:

SaveNewEvent
- ProjectEvent/AppendEvent
GetCurrentState

SaveNewEvent to the event table, at the same time update the existing or new created object table (eg. AccountTbl). And GetCurrentSate of current object table, which is the get operation.

Therefore, now under this pattern the workflow looks like:

Controller get/process request
Handler assemble the event from passed in request
Repository use Event Domain Model to reconstruct the command and update states.

Configuring Your Home Cluster Network: A Complete Guide

Cheedge Lee — Sun, 02 Mar 2025 20:19:17 +0000

It’s a renovated note.

Nowadays many of us will enjoy the cloud cluster rather than build a self managed cluster, as it’s less management, high availability, more secure, pay-as-you-go, and all the advantages you can think of the cloud computing. However, if you accidentally own several old computers, and don’t want to sell/transfer them and don’t know how to deal with them, a home-managed cluster will be a good choice. there is a lot of fun of a self managed cluster.

Architecture

Let’s define our architecture here: 4 nodes: 1 master/worker, 3 worker

1. master node

eth0: connect with institute cable
- public interface, using DHCP
- internet download, update
- users can access it: ssh/scp
eth1: connect with worker ndoes
- private interface, using static IP
- communicate other code:
  - ssh/scp
  - data transfer
  - parallel communicate

config network interface eth0 and eth1

# Configure public interface (assumes DHCP from institute network)
cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << EOF
TYPE=Ethernet
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
# Request static IP from our institute's DHCP server if possible
# This makes routing more reliable
DHCP_CLIENT_ID=cluster-master
EOF

# Configure private cluster network
cat > /etc/sysconfig/network-scripts/ifcfg-eth1 << EOF
TYPE=Ethernet
DEVICE=eth1
BOOTPROTO=static
IPADDR=192.168.10.1
NETMASK=255.255.255.0
ONBOOT=yes
EOF

# Apply new network configuration
systemctl restart network

NAT

# Enable IP forwarding persistently
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf

# Enable connection tracking timeout optimization for HPC workloads
echo "net.netfilter.nf_conntrack_tcp_timeout_established = 86400" >> /etc/sysctl.conf
echo "net.netfilter.nf_conntrack_max = 131072" >> /etc/sysctl.conf

# Apply sysctl changes
sysctl -p

# Set up NAT with higher connection limits
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.10.0/24 -j MASQUERADE

the last command is important, as it will contribute to the return traffic. We will explain later.

setup the firewall

# Clear existing rules
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X

# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP  # We'll explain this choice
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Allow established and related connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH from institute network
iptables -A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT

# Allow all traffic from the cluster's private network
iptables -A INPUT -i eth1 -s 192.168.10.0/24 -j ACCEPT

# Allow forwarding from cluster to internet
iptables -A FORWARD -i eth1 -s 192.168.10.0/24 -o eth0 -j ACCEPT

# Allow HTTP/HTTPS for package downloads
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 443 -j ACCEPT

# Set up local package caching repository later (optional)
iptables -A INPUT -i eth1 -p tcp --dport 80 -j ACCEPT

# Save iptables rules
iptables-save > /etc/sysconfig/iptables

I set the default FORWARD policy to DROP for security reasons:

It prevents unauthorized traffic from traversing the master node
It creates a default-deny stance, where only explicitly allowed traffic passes
It prevents potential lateral movement if one node is compromised

2. worker nodes

config

# Worker node 1 (192.168.10.2)
cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << EOF
TYPE=Ethernet
DEVICE=eth0
BOOTPROTO=static
IPADDR=192.168.10.2
NETMASK=255.255.255.0
GATEWAY=192.168.10.1
DNS1=8.8.8.8
ONBOOT=yes
EOF

# here GATEWAY will auto generate the route rule in table

# Worker node 2 (192.168.10.3)
# Change IPADDR=192.168.10.3 on the second worker node

# Worker node 3 (192.168.10.4)
# Change IPADDR=192.168.10.4 on the third worker node

# Restart network service on each worker node
systemctl restart network

Routing Configuration for Worker Nodes

As we use the master node eth1 (192.168.10.1) as the gateway for work nodes (GATEWAY=192.168.10.1), above setting creates a default route on each worker node that sends all traffic not destined for the local network (192.168.10.0/24) to the master node (192.168.10.1).

$ route -n
# see the result: (send all traffic (0.0.0.0/0) to gateway 192.168.10.1)
0.0.0.0         192.168.10.1      0.0.0.0         UG    0      0        0 eth0

Manual Command

Using manual command can also achieve the same result.

route add -net 0.0.0.0 gw 192.168.10.1

This manually adds a default route to the current routing table. It has the same immediate effect as the configuration file setting, but it's temporary and will be lost after a reboot or network service restart.

The difference is primarily in persistence and when the configuration happens. Using the network configuration file is the standard way to set up permanent routes in CentOS/RHEL systems.

Fire wall

# Clear existing rules
iptables -F
iptables -X

# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH from cluster nodes only
iptables -A INPUT -p tcp --dport 22 -s 192.168.10.0/24 -j ACCEPT

# HPC/MPI Communication - comprehensive approach
# Allow all TCP/UDP between cluster nodes for parallel computing
# this is optinal if you don't want do parallel computing
iptables -A INPUT -s 192.168.10.0/24 -p tcp -j ACCEPT
iptables -A INPUT -s 192.168.10.0/24 -p udp -j ACCEPT

# Save iptables rules
iptables-save > /etc/sysconfig/iptables
systemctl enable iptables

3. Package management

3.1 Local yum Repo

because the worker node will not have the access to internet, we keep them inside the private, therefore, for package installation, updating, we need to find a way to resolve these.
we want to specify the packages in our yum repo

so here we build a local yum repo

# On master node
# Install required packages
yum install -y createrepo nginx

# Create repository directory
mkdir -p /var/www/html/centos-repo

# Configure Nginx
cat > /etc/nginx/conf.d/repo.conf << EOF
server {
    listen 80;
    server_name _;
    root /var/www/html;

    location / {
        autoindex on;
    }
}
EOF

# Start and enable Nginx
systemctl enable nginx
systemctl start nginx

# Download packages to repository
yum install -y yum-utils
repotrack -p /var/www/html/centos-repo <package-name> 
# Repeat for packages we need

# Create repository metadata
createrepo /var/www/html/centos-repo

# Configure worker nodes to use this repository
cat > /etc/yum.repos.d/cluster-local.repo << EOF
[cluster-local]
name=Cluster Local Repository
baseurl=http://192.168.10.1/centos-repo
enabled=1
gpgcheck=0
EOF

3.2 Optional for package management

In order to realise

worker node package update
self-managed packages

we can also use scp to transfer package from master node.

# On master node, download and transfer RPM
yum install -y yum-utils
yumdownloader <package-name>
scp <package-name>.rpm 192.168.10.1:/tmp/

# On worker node
sudo rpm -ivh /tmp/<package-name>.rpm

3.3 directly route worker node to internet

If we don’t need high security, we can also open the private cluster to public internet. Which will configure the router table and we don’t discuss here.

4. Other Installations

ssh configuration

# On master node, generate SSH key
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""

# Copy the key to all nodes (including itself)
for i in {1..4}; do
  ssh-copy-id -i ~/.ssh/id_rsa.pub 10.10.10.$i
done

# Do the same on each worker node to allow any-to-any communication
# (Run similar commands on each worker node)

Network Monitor & iptables Log

# Install tools
yum install -y tcpdump nmap iftop

# Set up automatic monitoring with fail2ban to prevent brute force attacks
yum install -y fail2ban
cat > /etc/fail2ban/jail.local << EOF
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/secure
maxretry = 5
bantime = 3600
EOF

# Start and enable fail2ban
systemctl enable fail2ban
systemctl start fail2ban

# Add logging rules before the final DROP rules
iptables -A INPUT -j LOG --log-prefix "IPTables-Input-Dropped: " --log-level 4
iptables -A FORWARD -j LOG --log-prefix "IPTables-Forward-Dropped: " --log-level 4

# Save iptables rules
iptables-save > /etc/sysconfig/iptables

Optional Parallel Computing Configuration

# On all nodes (master and workers)
# Install OpenMPI
yum install -y openmpi openmpi-devel

# Configure environment in /etc/profile.d/
cat > /etc/profile.d/mpi.sh << EOF
export PATH=\$PATH:/usr/lib64/openmpi/bin
export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/usr/lib64/openmpi/lib
EOF

# Source the new environment
source /etc/profile.d/mpi.sh

# Test MPI communication
# Create a hostfile
cat > /home/username/hostfile << EOF
192.168.10.1 slots=128
192.168.10.2 slots=128
192.168.10.3 slots=128
192.168.10.4 slots=128
EOF

# Run a simple MPI test
mpirun -np 4 --hostfile /home/username/hostfile hostname

Torque/PBS

# Install Torque on master node
yum install -y torque-server torque-scheduler torque-client

# Configure server nodes file
cat > /var/torque/server_priv/nodes << EOF
192.168.10.1 np=128
192.168.10.2 np=128
192.168.10.3 np=128
192.168.10.4 np=128
EOF

# Start Torque server
systemctl enable pbs_server
systemctl start pbs_server

# Install Torque on worker nodes
for i in {2..4}; do
  ssh 192.168.10.$i "yum install -y torque-mom torque-client; systemctl enable pbs_mom; systemctl start pbs_mom"
done

5. Traffic Flow:

Forward Chain Traffic Flow in Both Directions

When we create the forward chain, iptables -A FORWARD -i eth1 -s 192.168.10.0/24 -o eth0 -j ACCEPT, this command let the traffic come into eth1 can be forward to eth0, which means traffic from worker nodes to master nodes, and master nodes forward it to institute internet. Here comes the question, where is the backward flow?

iptables -A FORWARD -i eth1 -s 192.168.10.0/24 -o eth0 -j ACCEPT

Let’s see how the traffic flows first.

Outbound Traffic (Worker → Internet)

The rule above allows packets to travel from the worker nodes (coming in on eth1) to be forwarded out to the institute network (through eth0). This handles the first half of any connection, which is the outbound request.

Return Traffic (Internet → Worker)

For the return traffic, typically we will think what we need as:

iptables -A FORWARD -i eth0 -d 192.168.10.0/24 -o eth1 -j ACCEPT

However, if we look closely at the original configuration, here is the command:

iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

This rule is handling the return traffic, because:

When a worker node initiates a connection, the outbound packet creates an entry in the connection tracking table (create a ESTABLISHED connection)
Any returning packets associated with that connection are marked as ESTABLISHED
The rule above allows all ESTABLISHED connections through, regardless of interface

This is more secure than explicitly allowing all traffic from eth0 to eth1, because it only permits return traffic for connections that were initiated from inside our cluster.

If this state tracking rule wasn't present, we would absolutely need to the backward traffic rule. Without either approach, connections would work one-way only, which means worker nodes could send requests, but never receive response…

Connection Flow

Let's trace a web request from a worker node:

Worker (192.168.10.2) tries to access google.com
Packet travels: Worker → Master's eth1
Master checks FORWARD chain, matches -i eth1 -s 192.168.10.0/24 -o eth0 rule
Master performs NAT, changing source IP to its own public IP
Packet leaves through eth0 to institute network
Google responds to master's public IP
Packet arrives at master's eth0
Master checks connection tracking table, sees this is a response
Packet is marked as ESTABLISHED
Master checks FORWARD chain, matches the ESTABLISHED rule
Master performs reverse NAT, changing destination to worker's IP
Packet leaves through eth1 to worker
Worker receives response

More details

more details can check the post before here.

DEV Community: Cheedge Lee

CKS Notes - Audit log

Audit workflow

Notice:

Policy

1. Resources you target

2. Which stages you log

3. Log level fields

3.4 Summary

CKS Notes - work node update

1. check the nodes we will upgrade

2. ssh to the node and check the version

3. ssh to work node, do the upgrade

4. back to controlplane and check the result

Cluster upgrade

CKS Notes - TLS

TLS

1. TLS config files

1.1 PKI files

1.2 Kubectl using TLS

~/.kube/config stores:

1.3 Application level (TLS Secrets)

1.3.1 TLS Secret and usage

2. TLS Flags

2.1 TLS Flags of apiserver

2.2 TLS Flags of etcd

Reference

CKS Notes - Image Security

ImagePolicyWebhook

1. External service

2. Admission Controller Configure file

3. KubeConfigFile

Example:

4. manifest kube-apiserver.yaml

Reference

Trivy Image scan

Reference:

CKS Notes - some notes on docker(podman)

1. Basic CMD

1.1 Tags

1.2 Layers (Caching)

1.3 share PID

2. Dockerfile Best Practice - Multi-stage build

3. Dockerfile Best Practice - Run as non-root user

4. Dockerfile Best Practice - Never echo secrets in Dockerfile

5. Docker related Security best practice

5.1 Do NOT expose Docker over TCP

5.2 Enforce TLS if you must expose TCP

5.3 Run dockerd as root (default), but avoid giving docker group to users

5.4 Use AppArmor or SELinux

5.5 Restrict container capabilities

5.6 Make /var/lib/docker read-only

5.7 Enable logging limits

CKS Notes - Pod Security

Layer A: Pod security settings

Layer B — Namespace-level Pod Security Admission (PSA)

work together

Best practice harden rules

0. Namespace

1. Run as non-root

2. Disallow privilege escalation

3. Drop unnecessary Linux capabilities

4. Use a read-only root filesystem

5. Use seccomp (system call filtering)

6. Avoid privileged containers

7. Use AppArmor (if available)

8. Restrict host access

Reference

CKS Notes - ServiceAccount

RBAC

Pod → Apiserver

1. Pod (no API server access)

2. Pod → API Server

Verify mount

Apiserver ←→ Kubelet (Pod)

Identities comparision

1. Human / Automation (kubectl, CI)

2. Kubelet (Node agent)

3. Pod (Workload)

Workflow

`~/.kube/config` stores:

5.6 Make `/var/lib/docker` read-only

1.2 `$HOME/.kube/config` (`admin.conf` copy)

1. check `--targets`:

2. check CIS version `--check`

1. on `master`

2. on `node`

`kubelet` is considering as node-level component

3. `etcd` check