DEV Community: ZachiNachshon

Install Traefik Ingress Controller in Kubernetes

ZachiNachshon — Sat, 01 Jan 2022 17:40:01 +0000

^{Credits: Logo by traefik.io}

Provide load balancing, SSL termination and name-based virtual hosting on a Kubernetes (k3s) cluster using Traefik ingress controller.

K8s Controller
- Prepare
- Install
- Uninstall
Traefik Dashboard
- Certificate
- Authentication
- Ingress
Demo Application
- Prepare
- Install
- Uninstall
- Certificate
- Ingress
Summary

Note: The domain referenced in this post is MY_DOMAIN, please change accordingly. If you interested in a local-only work mode, you don’t have to pay for a new domain, just decide on a name and use it. For example, if your desired domain is homelab.com, replace MY_DOMAIN with homelab.

Prerequisites

What is an Ingress Controller?
It is an API object that manages external access to a deployed service in a Kubernetes cluster, typically via HTTP/S. It provides load balancing, SSL termination and name-based virtual hosting.

Why would I want to use an Ingress Controller?
These are some of the immediate benefits using an ingress controller:

Simplify the way internal services interact with each other and re-route when required by changing only the Traefik routing rule.

Note: Instead of relying on an IP address that might change, simply define a hosted name e.g. serviceA.domain.com to address serviceA and use that on other internal services and/or from outside the cluster when in need to call serviceA.
Isn't serviceA.domain.com easier to remember than 192.169.200.xxx ?
Allows HTTPS traffic from outside the Kubernetes cluster while terminating encryption and allowing HTTP traffic between services within the cluster
Load balance traffic between services hosted outside the Kubernetes cluster

Note: Traefik is using custom cluster resource definitions. These CRDs gets installed via Traefik 2 Helm Chart. For CRDs schema click here.

K8s Controller

Traefik is a Kubernetes controller that manage the access to cluster services by supporting the Ingress specification. It receives requests on behalf of your system and finds out which components are responsible for handling them.

Prepare

We want Kubernetes to create the Traefik pod on the master node. In order to do that, we'll have to label that node and use nodeSelector attribute when installing Traefik Helm chart.

Get all nodes names and labels
```
kubectl get nodes --show-labels
```
Label kmaster node with node-type=master
```
kubectl label nodes kmaster node-type=master
```
Note: To remove a label, use the same command with dash after the label name e.g. kubectl label nodes kmaster node-type-."

Verify that label had been created successfully

kubectl get nodes --show-labels | grep node-type

Install

Create a traefik namespace
```
kubectl create namespace traefik
```

Add the containous Helm repository hosting the Traefik charts metadata

helm repo add traefik https://containous.github.io/traefik-helm-chart

Update local Helm chart repository cache
```
helm repo update
```

Search for latest traefik/traefik official Helm chart version

helm search repo traefik

# NAME              CHART VERSION   APP VERSION
# traefik/traefik   9.1.1           2.2.8

Install the Traefik Helm chart using the version from previous step

helm upgrade --install traefik \
    --namespace traefik \
    --set dashboard.enabled=true \
    --set rbac.enabled=true \
    --set nodeSelector.node-type=master \
    --set="additionalArguments={--api.dashboard=true,--log.level=INFO,--providers.kubernetesingress.ingressclass=traefik-internal,--serversTransport.insecureSkipVerify=true}" \
    traefik/traefik \
    --version 9.1.1

Verify installation

# Make sure all traefik deployed pods are running
kubectl get pods --namespace traefik

# Make sure custom resources *.traefik.containo.us were created successfully 
kubectl get crd | grep traefik

Uninstall

Remove traefik from the cluster

helm uninstall traefik --namespace traefik

Clear the traefik namespace
```
kubectl delete namespaces traefik
```

Traefik Dashboard

Dashboard is installed but disabled by default for security reasons. We will want to avoid using the kubectl proxy-forward option and allow the dashboard via HTTPS with proper TLS/Cert.

Certificate

We will create a certificate using cert-manager to allow accessing the Traefik dashboard via the hosted name traefik.MY_DOMAIN.com within our home network. Create a self signed certificate as described in here under traefik namespace.

Verify that a TLS secret had been created for the certificate:

kubectl get secret MY_DOMAIN-com-cert-secret --namespace traefik

Authentication

We will create a user / password basic authentication, please read here if you wish to use a different method.

Generate a flat-file that stores a username and password for basic authentication

bash <<'EOF'

# Change these credentials to your own
export TRAEFIK_UI_USER=admin
export TRAEFIK_UI_PASS=dashboard
export DESTINATION_FOLDER=${HOME}/temp/traefik-ui-creds

# Backup credentials to local files (in case you'll forget them later on)
mkdir -p ${DESTINATION_FOLDER}
echo $TRAEFIK_UI_USER >> ${DESTINATION_FOLDER}/traefik-ui-user.txt
echo $TRAEFIK_UI_PASS >> ${DESTINATION_FOLDER}/traefik-ui-pass.txt

htpasswd -Bbn ${TRAEFIK_UI_USER} ${TRAEFIK_UI_PASS} \
    > ${DESTINATION_FOLDER}/htpasswd

unset TRAEFIK_UI_USER TRAEFIK_UI_PASS DESTINATION_FOLDER

EOF

Create a Kubernetes secret based on the basic authentication file

kubectl create secret generic traefik-dashboard-auth-secret \
  --from-file=$HOME/temp/traefik-ui-creds/htpasswd \
  --namespace traefik

Ingress

Create an IngressRoute for accessing the dashboard, make sure to replace MY_DOMAIN with your domain name

cat <<EOF | kubectl apply -f -
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: traefik
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(\`traefik.MY_DOMAIN.com\`) && (PathPrefix(\`/api\`) || PathPrefix(\`/dashboard\`))
      services:
        - name: api@internal
          kind: TraefikService
      middlewares:
        - name: traefik-dashboard-auth # Referencing the BasicAuth middleware
          namespace: traefik
  tls:
    secretName: MY_DOMAIN-com-cert-secret
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-dashboard-auth
  namespace: traefik
spec:
  basicAuth:
    secret: traefik-dashboard-auth-secret
EOF

Check that resources were created successfully

# IngressRoute
kubectl describe ingressroute traefik-dashboard --namespace traefik

# Middleware
kubectl describe middleware traefik-dashboard-auth --namespace traefik

Check there are no error logs

kubectl logs -f $(kubectl get pods --namespace traefik | grep "^traefik" | awk '{print $1}') --namespace traefik

Set a hosted name traefik.MY_DOMAIN.com on a client machine (laptop/desktop)

# Edit the file using `sudo /etc/hosts`
# Append manualy to the existing k3s hosted name 
111.222.333.444 kmaster, traefik.MY_DOMAIN.com

# Alternatively, add a new hosted name entry with a one-liner
echo -e "111.222.333.444\ttraefik.MY_DOMAIN.com" | sudo tee -a /etc/hosts

Note: Replace 111.222.333.444 with the k3s master node IP address and MY_DOMAIN with your domain name.

Open browser at https://traefik.MY_DOMAIN.com/dashboard/

Note: If you encounter an untrusted certificate warning, please follow the cert-manager post Trust section for instructions on how to trust the traefik certificate.

Dashboard:

Demo Application

We will deploy an example whoami application that returns basic information about the client issuing the request. It'll allow us to check that Traefik was deployed successfully by addressing our demo application using whoami.MY_DOMAIN.com.

Prepare

Before deploying our application to the Kubernetes cluster, we will have to define which node(s) are eligible of assigning pods for it.

We will use the node-type=master label created at the Prepare step of the Traefik installation. To assign the whoami pod(s) to the kmaster node, we will use the nodeSelector attribute on a Kubernetes resource (Deployment / Pod) with that label.

Install

Create a playground namespace
```
kubectl create namespace playground
```

Create Kubernetes Deployment and Service with x2 running instances of the whoami application

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  namespace: playground
  labels:
    app: containous
    name: whoami
spec:
  replicas: 2
  selector:
    matchLabels:
      app: containous
      task: whoami
  template:
    metadata:
      labels:
        app: containous
        task: whoami
    spec:
      nodeSelector:
        node-type: "master"
      containers:
        - name: containouswhoami
          image: containous/whoami
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: playground
spec:
  ports:
    - name: http
      port: 80
  selector:
    app: containous
    task: whoami
EOF

Note: Please refer to the official docs to read more about Kubernetes Deployment or Service.

Verify both Deployment and Service were created successfully

# Deployment should indicate 2/2 running pods 
kubectl get deployments -n playground

# Service should indicate a cluster IP on port 80 without an external IP
kubectl get services -n playground

# Check that there are x2 running pods on kmaster node
kubectl describe pods whoami -n playground | grep "Status:"

Uninstall

Follow the installation instructions for Deployment, Service, IngressRoute, Middleware and execute both scripts while replacing the kubectl action:

# Instead of using apply
cat <<EOF | kubectl apply -f -  
...
EOF

# Replace with delete
cat <<EOF | kubectl delete -f - 
...
EOF

Clear the playground namespace
```
kubectl delete namespaces playground
```

Certificate

We will have to create a certificate using cert-manager to allow accessing the whoami application using the hosted name whoami.MY_DOMAIN.com within our home network. Create a self signed certificate as described in here under playground namespace.

Verify that a TLS secret had been created for the certificate:

kubectl get secret MY_DOMAIN-com-cert-secret --namespace playground

Ingress

Create ingress entry points to allow secure communication using HTTPS and allowing HTTP with HTTPS redirect middleware (make sure to replace MY_DOMAIN with your domain name)

cat <<EOF | kubectl apply -f -
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-https
  namespace: playground
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(\`whoami.MY_DOMAIN.com\`)
      kind: Rule
      priority: 10
      services:
        - name: whoami
          port: 80
  tls:
    secretName: MY_DOMAIN-com-cert-secret
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami
  namespace: playground
spec:
  entryPoints:
    - web
  routes:
    - match: Host(\`whoami.MY_DOMAIN.com\`)
      kind: Rule
      priority: 10
      services:
        # This IngressRoute will be never called due to the redirect middleware
        - name: whoami
          port: 80
      middlewares:
        - name: https-redirect
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: https-redirect
  namespace: playground
spec:
  redirectScheme:
    scheme: https
    permanent: true
EOF

Verify success status of HTTP routers and middlewares using the traefik dashboard

HTTP Routers:

HTTP Middlewares:

Set a hosted name whoami.MY_DOMAIN.com on a client machine (laptop/desktop)

# Edit the file using `sudo /etc/hosts`
# Append manualy to the existing k3s hosted name 
111.222.333.444 kmaster, traefik.MY_DOMAIN.com, whoami.MY_DOMAIN.com

# Alternatively, add a new hosted name entry with a one-liner
echo -e "111.222.333.444\twhoami.MY_DOMAIN.com" | sudo tee -a /etc/hosts

Note: Replace 111.222.333.444 with the k3s master node IP address and MY_DOMAIN with your domain name.

Open a web-browser at https://whoami.MY_DOMAIN.com and check that you can get a response from the service(s)

Note: Open http://whoami.MY_DOMAIN.com to check the HTTP -> HTTPS redirect.

Summary

Congratulations on completing a major part of deploying an ingress controller on your Kubernetes cluster ! 💪

What now? Now that you are familiar with how to use Traefik as an ingress controller, you can simplify the inter-services communication within the cluster and expose applications APIs / Web-UI and such via HTTPS in no time.

Please leave your comment, suggestion or any other input you think is relevant to this post in the discussion below.

Like this post?
You can find more by:

Checking out my blog: https://blog.zachinachshon.com
Following me on twitter: @zachinachshon

Thanks for reading! ❤️

Install Certificate Manager Controller in Kubernetes

ZachiNachshon — Sun, 02 May 2021 19:37:09 +0000

^{Credits: Logo by cert-manager.io}

Automate the process of issuing public key certificates from multiple sources, ensuring they are valid, up to date and renew before expiration.

K8s Controller
- Prepare
- Install
- Uninstall
Self Signed
- Issuer
- Certificate
- Secrets
- Trust
Advanced
- Share Secrets between Namespaces
Summary

Note: This post is a quick start guide for deploying and using cert-manager on a Kubernetes cluster.

Prerequisites

Why do we need to worry about certificates? When declaring a domain name i.e my-website.domain.com and addressing it from either internal network and/or public internet, the devices used to perform the call (web browsers, internal services, containers etc..) would require to check its validity. In order to do that, the domain name should have a certificate that is issued and trusted to operate securely.

Why do we need a certificate manager? Certificate validity has its expiration date, which means certificates have to get renewed. It might be a cumbersome task when there are many certificates to handle. This is the reason cert-manager exists, to help with issuing certificates from a variety of sources, such as Let’s Encrypt, a simple signing key pair or self signed. It will ensure certificates are valid, up to date and attempt to renew certificates at a configured time before expiry.

Note: The domain referenced in this post is MY_DOMAIN, please change accordingly. If you interested in a local-only work mode, you don't have to pay for a new domain, just decide on a name and use it. For example, if your desired domain is homelab.com, replace MY_DOMAIN with homelab.

K8s Controller

cert-manager is a Kubernetes controller that manage the certificate aspect of a cluster state. It looks after the state of certificates on a specific cluster and issue new ones or request to renew existing ones before expiration.

Prepare

Tip: This preparation step is relevant only if you wish to deploy the cert-manager pod on a specific cluster node such as master for example, otherwise please continue to the next Install step.

We want Kubernetes to create the cert-manager pod on the master node. In order to do that, we'll have to label that node and use nodeSelector attribute when installing cert-manager Helm chart.

Get all nodes names and labels
```
kubectl get nodes --show-labels
```
Label kmaster node with node-type=master
```
kubectl label nodes kmaster node-type=master
```
Note: I'll refer to the master node named kmaster as specified in the RPi cluster installation post, replace the name with the one you have assigned to the master node on your Kubernetes cluster.

Verify that label had been created successfully

kubectl get nodes --show-labels | grep node-type

Install

Create a cert-manager namespace
```
kubectl create namespace cert-manager
```
Disable resource validation on the cert-manager namespace
```
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
```
Note: cert-manager deploys a webhook component to perform resource validations on Issuer, ClusterIssuer and Certificate. This webhook shouldn't run on the same namespace the cert-manager is deployed on.

Add the required Helm repository

helm repo add jetstack https://charts.jetstack.io

Update your local Helm chart repository cache
```
helm repo update
```

Search for latest jetstack/cert-manager official Helm chart version

helm search repo cert-manager

# NAME                      CHART VERSION     APP VERSION
# jetstack/cert-manager     v1.2.0            v1.2.0

Install the cert-manager Helm chart using the version from previous step

Without node affinity:

helm upgrade --install cert-manager \
    --namespace cert-manager \
    --version v1.2.0 \
    jetstack/cert-manager \
    --set installCRDs=true

With node affinity (if you have followed the Prepare step):

helm upgrade --install cert-manager \
    --namespace cert-manager \
    --version v1.2.0 \
    --set nodeSelector.node-type=master \
    --set webhook.nodeSelector.node-type=master \
    --set cainjector.nodeSelector.node-type=master \
    jetstack/cert-manager \
    --set installCRDs=true

Note: Make sure you are running on Kubernetes > v1.19 to have the installCRDs flag available.

Verify installation

# Make sure all cert-manager deployed pods are running
kubectl get pods --namespace cert-manager

# Make sure custom resources *.cert-manager.io were created successfully 
kubectl get crd | grep cert-manager

# Verify that ClusterIssuer is non-namespaced scoped ('false')
# so it can be used to issue Certificates across all namespaces
kubectl api-resources | grep clusterissuers

Uninstall

Remove cert-manager from the cluster

helm uninstall cert-manager --namespace cert-manager

Clear the namespace
```
kubectl delete namespaces cert-manager
```

Self Signed

What? The self signed issuer does not represent a certificate authority as such, but instead denotes that certificates will be signed through “self signing” using a given private key.

Why? This means that the provided private key of the resulting certificate will be used to sign its own certificate.

When? This Issuer type is useful for bootstrapping the CA certificate key pair for some Private Key Infrastructure (PKI), or for otherwise creating simple certificates. Clients consuming these certificates have no way to trust this certificate since there is no CA signer apart from itself, and as such, would be forced to trust the certificate as is.

Who? Clients consuming these certificates could be services deployed to our local Kubernetes cluster that are exposed from outside the cluster such as Kubernetes dashboard, Jenkins UI, Private Docker Registry UI etc..

Issuer

Create a certificate ClusterIssuer
```
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: MY_DOMAIN-ca-issuer
spec:
  selfSigned: {}
EOF
```
Important: cert-manager issues certificates through an Issuer. The Issuer can issue certificates for the namespace it is created on, but a ClusterIssuer can create certificates for any namespace.

Wait for status to become Ready

kubectl get clusterissuers MY_DOMAIN-ca-issuer -o wide

Certificate

When creating a new certificate, make sure to create one on a named namespace. It'll get verified by cert-manager even-though it exists on a different namespace since we're referencing a ClusterIssuer.

Tip: Replace MY_NAMESPACE with the namespace of your choise.

Create a namespaced X.509 certificate (check here for official schema)

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: MY_DOMAIN-com-cert
  namespace: MY_NAMESPACE
spec:
  secretName: MY_DOMAIN-com-cert-secret
  isCA: true
  commonName: '*.MY_DOMAIN.com'
  organization:
    - MY_DOMAIN
  dnsNames:
    - MY_DOMAIN.com
    - '*.MY_DOMAIN.com'
  keySize: 2048
  keyAlgorithm: rsa
  issuerRef:
    name: MY_DOMAIN-ca-issuer
    kind: ClusterIssuer
EOF

Check certificate status is Issued

kubectl describe certificate MY_DOMAIN-com-cert --namespace MY_NAMESPACE

Check that secret MY_DOMAIN-com-cert-secret was created successfully

kubectl get secret --namespace MY_NAMESPACE
kubectl get secret MY_DOMAIN-com-cert-secret -o yaml --namespace MY_NAMESPACE

(Optional): When in need to delete a certificate

# Delete certificate
kubectl delete certificate MY_DOMAIN-com-cert --namespace MY_NAMESPACE

# Delete the auto generated secret
kubectl delete secret MY_DOMAIN-com-cert-secret --namespace MY_NAMESPACE

Secrets

Follow these instructions to export the cert-manager generated certificate secrets as local files.

Note: Sometimes there is a need to use the secret values from outside the Kubernetes cluster. Such example is setting the cert_file as a trusted certificate on a client machine (laptop / desktop), specially when using a self signed certificate.

Create a local destination folder

mkdir -p $HOME/temp/MY_NAMESPACE/cert-secrets
export MY_DOMAIN=<insert-domain-name-here>
export MY_NAMESPACE=<insert-namespace-here>

Export the certificate secrets

cert_file - client certificate path used for authentication

kubectl get secret ${MY_DOMAIN}-com-cert-secret \
   --namespace ${MY_NAMESPACE} \
   -o jsonpath='{.data.tls\.crt}' | base64 -D \
   > $HOME/temp/${MY_NAMESPACE}/cert-secrets/cert_file.crt

key_file - client key path used for authentication

kubectl get secret ${MY_DOMAIN}-com-cert-secret \
   --namespace ${MY_NAMESPACE} \
   -o jsonpath='{.data.tls\.key}' | base64 -D \
   > $HOME/temp/${MY_NAMESPACE}/cert-secrets/key_file.key

ca_file - CA certificate path used to verify the remote server cert file

kubectl get secret ${MY_DOMAIN}-com-cert-secret \
   --namespace ${MY_NAMESPACE} \
   -o jsonpath='{.data.ca\.crt}' | base64 -D \
   > $HOME/temp/${MY_NAMESPACE}/cert-secrets/ca_file.crt

Check that x3 secrets exported successfully

ls -lah $HOME/temp/${MY_NAMESPACE}/cert-secrets

Clear exported variables
```
unset MY_DOMAIN MY_NAMESPACE
```

Trust

When addressing a Kubernetes ingress controller resource that had been signed with a self signed certificate secret, clients such as web-browsers would warn us of an invalid certificate authority or invalid certificate.

ERR_CERT_AUTHORITY_INVALID

ERR_CERT_INVALID

Since we are the ones that created the certificate we would like to tell our clients (laptop/desktop) to trust it. This is how you trust a certificate on macOS, for other operating systems please refer to official documentation.

Open the certificate file in macOS Keychain Access

CLI: open $HOME/temp/${MY_NAMESPACE}/cert-secrets/cert_file.crt
Other: double click on the cert_file.crt

Double click on the *.MY_DOMAIN.com certificate and expand the Trust section
Within the When using this certificate select Always Trust
Open a web-browser and navigate to one of the domain names using the self signed certificate and verify that there are no errors

Advanced

This section contains non conventional tip & tricks, use it cautiously.

Share Secrets between Namespaces

When? Use when in need to copy a cert-manager generated certificate secret to a different namespace than the one it was created on.

Why? Useful when you need to reference the secret in some kind of ingress controller under its authentication spec which exists on a different namespace.

In order to solve this limitation we simple need to copy the secret to a different namespace as follows:

kubectl get secret MY_DOMAIN-com-cert-secret -n SOURCE_NAMESPACE -o yaml \
| sed s/"namespace: SOURCE_NAMESPACE"/"namespace: DESTINATION-NAMESPACE"/\
| kubectl apply -n DESTINATION-NAMESPACE -f -

Important: It is not advised to copy secrets manually since cert-manager would lose track of the secret as it was copied manually and when the time to renew is due, the secret would become invalid.

Summary

This post was a sneak peak on how to create certificates and manage their state using cert-manager Kubernetes controller.

What now? Now that you have a local self signed certificate, you can continue to create an ingress controller and expose internal web services within your home network such as Kubernetes dashboard, Jenkins UI, your web application UI with their own virtual hosted names i.e. kubernetes.MY_DOMAIN.com, jenkins.MY_DOMAIN.com, mydashboard.MY_DOMAIN.com etc..

Please leave your comment, suggestion or any other input you think is relevant to this post in the discussion below.

Like this post?
You can find more by:

Checking out my blog: https://blog.zachinachshon.com
Following me on twitter: @zachinachshon

Thanks for reading! ❤️

Installing Helm, Kubernetes Package Manager

ZachiNachshon — Sun, 02 May 2021 07:22:28 +0000

^{Credits: Logo by helm.sh}

Learn how to configure the deployment of multiple Kubernetes resources as a single unit using Helm package manager.

Helm CLI
- Install
- Uninstall
Charts
- Search
- Install
- Upgrade
Summary

Note: This post is a TL;DR for sharing basic Helm v3 usage which is required for the reader just to get started, for additional information please follow the official docs.

Helm-CLI

Mostly TL;DRs for installing the Helm CLI.

Install

Install using your favorite package manager, consult with the official docs for additional information.

# tl;dr for macOS
brew install helm

Uninstall

Follow the removal instructions of the package manager you've used previously.

# tl;dr for macOS
brew uninstall helm

Charts

Helm Charts are the actual "packages" managed by Helm. A chart is a just another YAML file which consists of custom attributes that eventually gets converted into Kubernetes YAML resources and deployed into the cluster.

Why? Instead of managing multiple resources on our own and their respective Kubernetes YAML resource files, we can wrap them all into a single Helm chart and address them as a unit, meaning, install/update/uninstall using a single command. Charts can go from a simple hello-world application to a fully blown web application i.e servers, databases, cache etc..

Note: On the following sections, I'll use traefik proxy as an example Helm chart.

Search

In order to be able to search for charts we'll have to find its respective Helm repository that returns its metadata. Search in artifacthub.io for the Helm repository for the chart that you wish to use.

We'll add the containous Helm repository hosting the traefik charts metadata
```
helm repo add traefik https://containous.github.io/traefik-helm-chart
```
Update local Helm chart repository cache
```
helm repo update
```

Search for a specific chart by name

helm search repo traefik

# NAME              CHART VERSION   APP VERSION
# traefik/traefik   9.1.1           2.2.8

Install

Install a Helm chart directly from the command line. For additional installation options read here.

helm install traefik \
    --namespace traefik \
    traefik/traefik \
    --version 9.1.1

Note: It is possible to use the same Helm command for both install & upgrade by supplying the flag --install. If a release doesn't already exist by the supplied name, an install command would trigger.
Replace helm install with helm upgrade --install.

Upgrade

Upgrade or override existing values of the previously deployed chart with a new chart release. For additional upgrade options read here.

helm upgrade traefik \
    --namespace traefik \
    --set="additionalArguments={--log.level=DEBUG}" \
    traefik/traefik \
    --version 9.1.1

Note: When updating partial attributes on a pre-installed Helm chart, the Kubernetes resource pods which relates to these specific attributes would get created anew to apply the new changes.

Summary

This post is just the tip of the iceberg regarding Helm charts possibilities, please refer to the official Helm docs for further reading.

Please leave your comment, suggestion or any other input you think is relevant to this post in the discussion below.

Like this post?
You can find more by:

Checking out my blog: https://blog.zachinachshon.com
Following me on twitter: @zachinachshon

Thanks for reading! ❤️

Install Rancher K3s on Raspberry Pi Cluster

ZachiNachshon — Mon, 05 Apr 2021 19:55:23 +0000

^{Credits: Logo by cncf-branding}

Install a Rancher Labs Kubernetes distribution (k3s) on a Raspberry Pi cluster.

Master Server
- Install
- Uninstall
Worker Node
- Join a Cluster
- Uninstall
Utilities
- kubectl
- k9s
Summary

Note: This post refers to laptop / desktop as client machines. These are the clients used to connect to the Raspberry Pi master / worker nodes remotely.

Prerequisites

Setting Up a Raspberry Pi Cluster

Master Server

What is a Kubernetes master node? A master node is a server that controls and manages a set of worker nodes, in our case it is the Raspberry Pi that controls the rest of the Raspberry Pi(s) on our cluster.

Install

SSH into the Raspberry Pi server that is intended to operate as the Kubernetes master. It should be the one named kmaster as instructed by this post
```
# Connect to RPi server that operates as the k8s master
ssh pi@kmaster
```
Run the following command to install a plain version of k3s without traefik load balancer and k8s-dashboard
```
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=" --no-deploy traefik --no-deploy kubernetes-dashboard"  sh -
```
Note: We will install a plain version of k3s without Traefik load balancer and/or Kubernetes dashboard. These should be covered by other dedicated blog posts.

Verify that k3s was installed successfully. Run the following commands from within the RPi master server

# Check for status - active (running)
sudo systemctl status k3s

# Check for status - Ready
sudo kubectl get nodes

# Optional - check that there are no error logs
tail -f /var/log/syslog

(Optional): Run when in need to restart k3s
```
# Restart k3s
sudo systemctl restart k3s
```
Note: The k3s service is automatically started and restarted during installation. The install script will install k3s and additional utilities, such as kubectl, crictl, k3s-killall.sh, and k3s-uninstall.sh.

Note: During installation kubectl on the master server will be aliased to the command k3s kubectl so that we can use the pre-packaged version of kubectl. k3s uses a container runtime called containerd directly (no docker), interact using crictl.

Uninstall

SSH into the k3s master server
```
ssh pi@kmaster
```
Uninstall k3s by executing the following script:
```
/usr/local/bin/k3s-uninstall.sh
```
Note: Rancher k3s service configuration can be found at /etc/rancher/k3s/k3s.yaml.

Note: Container runtime containerd configuration can be found at /var/lib/rancher/k3s/agent/etc/containerd/config.toml.

Worker Node

What is a Kubernetes worker node? These are Raspberry Pi servers that act as workload runtimes i.e. run our applications, jobs, whatever we require them to run but they aren't the ones that manage the cluster, just the ones that "get the job done".

Join a Cluster

Extract the k3s join cluster token

# SSH to the RPi master server
ssh pi@kmaster

# Extract the join token
sudo cat /var/lib/rancher/k3s/server/node-token

# Alternatively, you can run this on-liner directly from a client machine
ssh pi@kmaster "sudo cat /var/lib/rancher/k3s/server/node-token"

Find the k3s master server IP address that is assigned to kmaster, either from the server itself or from a client machine if you've followed this post

# From the RPi master server
ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1

# Alternatively, from a client machine
cat /etc/hosts | grep kmaster | awk '{print $1}'

SSH into a Raspberry Pi server intended to be used as a Kubernetes worker node. It would be the one named knode<number> as instructed on this post
```
# Connect to a k3s worker node
ssh pi@knode<number>  
```

Run the following command to install k3s-agent and join the worker node to an existing cluster

# Replace MASTER-IP-ADDRESS with the master server IP address from previous step
# Replace JOIN-TOKEN with the join token from previous step
curl -sfL http://get.k3s.io | K3S_URL=https://<MASTER-IP-ADDRESS>:6443 \
K3S_TOKEN=<JOIN-TOKEN> sh -

Verify that k3s-agent was installed successfully. Run the following commands from within the RPi worker server

# Check for status - active (running)
sudo systemctl status k3s-agent

# Optional - check that there are no error logs
tail -f /var/log/syslog

Repeat the above steps for every Raspberry Pi board intended to be used as a Kubernetes worker node

Uninstall

SSH into the k3s worker node
```
ssh pi@knode<number>
```
Uninstall k3s-agent by executing the following script:
```
/usr/local/bin/k3s-agent-uninstall.sh
```

Utilities

These are common utilities that should get installed on client machines to interacts with k3s master server.

Why do I need to install them?
You'll want to interacts with Kubernetes in order to deploy services, execute Helm charts and/or use utilities that grant you cluster visibility.

Note: If you are planning to interact with k3s on a CI environment, make sure that the agent image you are using in the pipeline includes utilities such as kubectl.

kubectl

Install kubectl, a command-line-interface tool that allows you to run commands against a remote k3s cluster.

On a client machine, create a new empty k3s config file

mkdir -p $HOME/.kube/k3s 
touch $HOME/.kube/k3s/config
chmod 600 $HOME/.kube/k3s/config  # Set limited user permissions

Copy the k3s cluster configuration from the RPi master server

ssh pi@kmaster "sudo cat /etc/rancher/k3s/k3s.yaml" > $HOME/.kube/k3s/config

Edit the k3s config file on the client machine and change the remote IP address of the k3s master from localhost/127.0.0.1 to kmaster

# Edit master config
vim $HOME/.kube/k3s/config

# Search for the 'server' attribute located in - 
# clusters:
# - cluster:
#   server: https://127.0.0.1:6443 or https://localhost:6443
#
# Change 'server' value to https://kmaster:6443

Note: Make sure kmaster is properly defined as a host name in /etc/hosts, otherwise - use the RPi master server IP address.

Install kubectl as described in the official docs

# tl;dr - macOS only
brew install kubectl

# Verify client version
kubectl version --client

Export k3s config file path as KUBECONFIG environment variable and by doing that set the kubectl context to use the RPi k3s cluster
```
export KUBECONFIG=$HOME/.kube/k3s/config
```
Note: Add the export command into a .bash_profile / .bashrc file. This way every new shell session would have the k3s cluster config set as the kubectl active context. Optional: check this post to manage your dotfiles in style.

Verify that kubectl was installed properly and can communicate with the RPi master server

kubectl get nodes

# Expect the following respones as success:
#
# NAME      STATUS   ROLES                  AGE   VERSION
# knodeX    Ready    <none>                 10m   v1.20.4+k3s1
# ...
# knode1    Ready    <none>                 23m   v1.20.4+k3s1
# kmaster   Ready    control-plane,master   52m   v1.20.4+k3s1

(Optional): read here for additional information about kubectl

k9s

Install k9s, a terminal UI that interacts with the k3s cluster, increase velocity by saving you from typing repetitive commands and/or the need to alias common ones. It allows easy navigation, observation and management - all in one package.

Install as instructed on the official repository docs
```
# tl;dr for macOS
brew install k9s
```
Note: Make sure $KUBECONFIG is properly defined and set to k3s config path.
In case you are working on a single cluster and it is the default one, move to the next step, otherwise, if you are working on multiple clusters and/or using a cluster which isn't named default, change the currentContext and currentCluster attributes on the k9s config file to the proper cluster values.

Note: k9s configuration file can be found at $HOME/.k9s/config.yml
Run k9s on a fresh shell session and verify that you can connect the k3s cluster successfully
(Optional): read here for additional information about k9s

Summary

Well done for successfully installing a Kubernetes cluster on top of your Raspberry Pi cluster ! 👏

What now? Check back for future posts explaining how to install a load balancer, certificate manager and a private docker registry on that cluster.

Please leave your comment, suggestion or any other input you think is relevant to this post in the discussion below.

Like this post?
You can find more by:

Checking out my blog: https://blog.zachinachshon.com
Following me on twitter: @zachinachshon

Thanks for reading! ❤️

Setting Up a Raspberry Pi Cluster

ZachiNachshon — Sun, 28 Feb 2021 20:18:55 +0000

^{Credits: Logo created by raspberrypi.org}

Installation instructions for setting up a local Raspberry Pi cluster at your home desk.

Operating System on SD Card
Network Setup
RPi Configuration
SSH Config
- Client
- RPi Server
Verification
Troubleshooting
Summary

Note: For the time this post was published, there isn't an official Raspberry Pi OS image that supports a 64 bit system. There are beta versions but with limitations. This blog post won't cover them until officially released.

Operating System on SD Card

Connect the SD card to your local machine (laptop / desktop)
Download Raspberry Pi OS Lite (Debian Buster) image from here
Flash OS image using Raspberry Pi Imager as specified in here
After flashing, eject and re-connect the SD card again to your local machine
Check for boot connected device
Mount the SD card

Open a terminal session and create a text file named ssh in the boot partition

  # If you are working on macOS
  sudo touch /Volumes/boot/ssh

  # Other Unix-like operating system
  sudo touch <path-to-rpi-boot-volume>/ssh

Eject the SD card and connect it to the Raspberry Pi

Network Setup

Power up the RPi and connect directly to the home router
Open the router network dashboard

Note: Your router dashboard address might differ, check with your router manufacture guide.
Verify under LAN settings -> Client List that there is a new raspberrypi client
Assign a static IP address using this guide

RPi Configuration

SSH into the server using ssh pi@<RPI-IP-ADDRESS> with an IP address from previous step and the default password raspberry
Set the GPU memory split to 16MB by editing the RPi configuration file sudo vi /boot/config.txt and appending gpu_mem=16 to it

Note: We specify the minimum memory possible to reserve for GPU (display) since we won’t require user interface.
Reboot using sudo reboot for changes to take effect
SSH again to the RPI server and type sudo raspi-config. Edit the following settings:
- Change the password for the pi user (recommended)
- Set the hostname to your liking
- Make sure SSH server is enabled
  
  Note: If you are planning on using this server as a Kubernetes master - name it kmaster. If it is a Kubernetes node, name it knode<number> while <number> is the next in-line number of your cluster stack.
Close the SSH session and reconnect to the RPi server again
Verify hostname was properly set and force manual replacement, if required. From the RPi terminal:
- Run cat /etc/hostname and check for kmaster
- Run cat /etc/hosts and check for 127.0.1.1 kmaster
- Run hostname which should return kmaster, otherwise run sudo hostname kmaster and check again
Note: Use kmaster or knode<number> according to the server type if you are setting up a Kubernetes cluster.

(Optional): Install your favourite utilities on the RPi server:

  # Use Vim as a text editor
  sudo apt-get -y install vim

(Optional): Set your preferred aliases on ~/.bashrc:

  # Open bash run command file for editing
  vim ~/.bashrc

  # List all files/directories including hidden ones with size unit suffixes 
  alias l="ls -lah"

(Optional): If you are planning to install the Rancher k3s version of Kubernetes, you should enable a few container features by adding them to the end of the /boot/cmdline.txt file:
```
  # Edit file with sudo
  sudo vim /boot/cmdline.txt

  # Append the following
  cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
```

SSH Config

We need to configure secure shell access for client <-> RPi server communication. It will allow us to access the RPi server from client machines such as our laptop and also allow secure communication between the RPi server and our locally installed utilities.

Client

These instructions are relevant to the computer being used to connect to the RPi server:

Create directory ~/.ssh if it doesn't exists and cd into it
Run ssh-keygen (with name kmaster OR knode<number>, no passphrase)
Add the private key kmaster to the ssh agent (select between permanent/temporary)
```
ssh-add ~/.ssh/kmaster      # Add temporary to keychain
ssh-add -K ~/.ssh/kmaster   # Add permanently to keychain
```
Note: This will allow a secure communication without prompting for the password every time.
Copy the public key to RPi server
```
  # Master node
  ssh-copy-id -i ~/.ssh/kmaster.pub pi@<RPI-IP-ADDRESS>

  # Agent node
  ssh-copy-id -i ~/.ssh/knode<number>.pub pi@<RPI-IP-ADDRESS>
```
Important: For this step you will need to authenticate with your password. A file named ~/.ssh/authorized_keys is auto-created with the public key content on the RPi server.
(Optional): If you have defined a static IP for the RPi server as described in here, add named hosts records on your client machine:

   # Use names instead of IP addresses
   echo -e "<MASTER-IP-ADDRESS>\tkmaster" | sudo tee -a /etc/hosts
   echo -e "<NODE-IP-ADDRESS>\tknode1" | sudo tee -a /etc/hosts

RPi Server

These instructions are intended for the RPi server to enable SSH communication:

Disable SSH PasswordAuthentication:
- Edit ssh_config by sudo vim /etc/ssh/ssh_config
- Uncomment PasswordAuthentication by removing the # prefix
- Change its value to no
- Make sure PasswordAuthentication is properly aligned (4 spaces)
Restart SSH server

   sudo /etc/init.d/ssh restart
   sudo /etc/init.d/ssh status   # Verify SSH server is running

Verification

To verify everything is set-up correctly, try to connect from the client machine i.e. laptop to the RPi server with the following command:

ssh pi@kmaster
ssh pi@knode<number>

Troubleshooting

Locale

What? Locale errors/warnings when connecting to a RPi server and/or running locale on a server node. These are a few example errors:

setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

Solution: We'll set our locale to en_US by running the following steps:

SSH to the RPi server
Edit the locale.gen file using sudo vi /etc/locale.gen
Uncomment the line with en_US.UTF-8 by removing the # character (make sure there are no leading spaces)
Run sudo locale-gen en_US.UTF-8
Run sudo update-locale en_US.UTF-8
Run locale and make sure there are no errors/warnings

Summary

By the end of this post you should have a working headless (non-GUI) Raspberry Pi(s) connected to your home network with SSH communication available, good job ! 👏

What now? You are welcome to check back for a future blog post on how to install Kubernetes on top of your amazing RPi cluster.

Please leave your comment, suggestion or any other input you think is relevant to this post in the discussion below.

Like this post?
You can find more by:

Checking out my blog: https://blog.zachinachshon.com
Following me on twitter: @zachinachshon

Thanks for reading! ❤️

Configure Local Environment Using Dotfiles

ZachiNachshon — Fri, 19 Feb 2021 08:32:26 +0000

^{Credits: Logo created by Joel Glovier}

Automate your local dev configuration in style on a personal, work or any other machine using a dedicated CLI utility.

TL;DR

Using a dotfiles repository has never been so easy. Interact with any dotfiles repository using a well defined CLI utility rather than executing random scripts.

Control the following items from a Git backed repository:

.dotfiles (aliases, exports, functions, shell RC files and others...)
Settings and configurations (.gitconfig, .vimrc and others...)
Homebrew (packages, casks, services, drivers)
macOS/Linux settings and preferences

LONG

Incentive
Solution
What's included?
Homebrew
dotfiles
- Session Folder
- Custom Folder
- Home Folder
- Transient Folder
- Shell Folder
macOS/Linux settings override
Demo
Summary

What is a .dotfile?

It is a standard text file usually located within the $HOME folder. It contains a dot prefix in its name and is a hidden file which you can list with ls -a on a *nix system.

What is a Homebrew package / cask?

Homebrew is a package manager for macOS, it allows you to install CLI utilities (packages) & GUI applications (casks).

Why override macOS settings?

It is relevant if you have custom macOS settings. These are usually the settings that you change from the macOS GUI (Graphical User Interface) or from the terminal. You should back these custom settings so you'll be able to restore them on every new/formatted machine.

Incentive

We have all been there, being lost in the sea of information on our constant learning experience on a daily basis. When we identify a command / script / keybinding that it is useful to us, we'll add it to a note taking application hoping to remember where it is stored for the next time.

The same goes for local applications and CLI utilities. When receiving a new machine or going through an OS re-install, we start a manual installation process of all the applications and CLI utilities we are used to, hopefully not to miss a few.

And what about run command files (*rc) such as zshrc and bashrc for example. They usually contain custom content such as terminal colors, plugins etc.. which you'll want to restore so you'll feel right at home within your comfortable shell.

It's not a simple task trying to keep our magic scripts that are scattered all over the place, either if its changing a system setting we've got used to or the repetitive command we're constantly searching for in Google since we cannot find the time to add a snippet for it since we cannot seem to find the appropriate note-taking-application-to-rule-them-all.

Solution

Automate your local development environment installations/updates with just a few terminal commands. Manage everything under a Git backed repository so you'll keep your setting in a centralized location, detached from any machine and enjoy all the benefits a version control has to offer.

How does keeping dotfiles under a GitHub repo helpful?

Even though the dotfiles are located under the folder you have cloned the Git repository into, by using symlinks (symbolic link / soft link) we're able to link them to $HOME or any other folder and every change within the repository is reflected on every new terminal shell session since these files get sourced with their latest content.

Tip: Additionally, you can call dotfiles reload on the existing session without the need to create a new session.

How do we start?

There are a lot of dotfiles repositories publicly available in GitHub. People share their own managed dotfiles, either if it's a single script you'll have to add to the .bash_profile or other more complex alternatives that require a different setup.

In this blog post I'll share an example dotfiles solution similar to the one I'm using which is managed by a dotfiles-cli utility that allows its users some ease of mind when it comes to local environment setup.

Please feel free to fork my dotfiles GitHub repository and add your own content on top of it !

In case you wish to check the repository out before forking, you are welcome to clone and change it to suite your needs:

$ git clone https://github.com/ZachiNachshon/dotfiles-example.git ~/<my-codebase-folder>

Why using a dotfiles CLI utility?

Simplify the complex dotfiles repository wiring by separating the files from the management layer
Use a dedicated CLI utility to control all aspects of the dotfiles repository with ease
Having a coherent dotfiles structure that is easy to get familiar with
Allow a generic CLI to control multiple dotfiles repositories (private and public)
Avoid from running arbitrary scripts

How to install the dotfiles CLI?

Install dotfiles-cli via Homebrew, pre-built release or from sources. Please head over to the download section for further installation information.

What's included?

A well organized folder layout containing categorized scripts with minimum overhead to glue them all together using the dotfiles-cli utility.

An example of an expected repository structure:

   .
   ├── ...
   ├── brew   
   │   ├── casks.txt
   │   ├── drivers.txt
   │   ├── packages.txt
   │   ├── services.txt
   │   └── taps.txt
   │
   ├── dotfiles               
   │   ├── custom  
   │   │   ├── .my-company  
   │   │   └── ...
   │   ├── home
   │   │   ├── .gitconfig       
   │   │   ├── .vimrc
   │   │   └── ...
   │   ├── session 
   │   │   ├── .aliases
   │   │   └── ...
   │   ├── shell
   │   │   ├── .zshrc
   │   │   └── ...
   │   └── transient
   │       └── .secrets
   │
   ├── os
   │   ├── linux
   │   │   ├── key_bindings.sh
   │   │   └── ...
   │   └── mac
   │       ├── finder_settings.sh  
   │       └── ...
   │
   ├── plugins
   │   ├── zsh
   │   │   ├── oh_my_zsh.sh  
   │   │   └── ...
   │   └── bash 
   │       ├── dummy.sh
   │       └── ...
   └── ...

Tip: Run dotfiles structure to get a reminder on the expected dotfiles repo structure.

Homebrew

Automate the installation process of favorite CLI utilities & GUI applications by using Homebrew macOS package manager. All you will have to do is to update simple text files with the package, cask, driver or service you wish to install. Installation and updates takes place whenever dotfiles brew <option> command is being called.

To add a CLI utility (a.k.a Homebrew package):

Search for a desired package in here
Add the package name to the packages.txt file
Run dotfiles brew packages to install/update

To add a GUI application (a.k.a Homebrew cask):

Search for a desired cask in here
Add the cask name to the casks.txt file
Run dotfiles brew casks to install/update

Tip: Similar flow for installing drivers and services via Homebrew.

dotfiles

Session Folder

A folder containing dotfiles which are relevant across all machines (work, personal etc..) that get sourced along on every new shell session or upon dotfiles reload command.

Example of such session files I use (you can add your own):

.aliases - define shortcuts for commonly used *nix commands

Example:

   # -=-= Custom =-=-
   alias l="ls -lah" 
   alias dl="cd ~/Downloads"
   alias gop="cd ${GOPATH}/src/github.com"

   # -=-= ZSH =-=-
   alias zshconfig="vim ~/.zshrc"

.functions - add new utility scrips to the functions collection so they'll be available right away

Example:

   # Create a new directory and cd into it
   mkd() {
     mkdir -p "$@"
     cd "$@" || exit
   }

.paths - add new environment variables you wish to make available on new shell sessions

Example:

   # -=-= Kubernetes =-=-
   export KUBECONFIG=$HOME/.kube/k3s/config

   # -=-= Path =-=-
   export PATH=/usr/local/bin:/usr/bin:${JAVA_HOME}/bin:${GOROOT}/bin:${GOPATH}/bin:${PATH}

Custom Folder

A folder containing custom files to symlink, usually relevant to specific machines e.g. work related or personal etc..

I have added an example file to the repository (you can and should add your own):

.my-company - a work related content that should get sourced on every new shell session

Example:

   alias work-repos="cd codebase/my-company-repos" 

   work() {
      echo "I work at ACME Corporation."
   }

Home Folder

A folder containing files that should get symlinked from the dotfiles GitHub repository to the $HOME folder for the same reasons we have described in here. A few examples are .gitignore_global, .gitconfig, .vimrc and others...

To add such file as described:

Place it within <repo-root>/dotfiles/home/
Run dotfiles sync home

Transient Folder

A folder containing transient files, meaning, files that shouldn't get checked-in into source control or symlinked anywhere but should get sourced along on every new shell session.

You can use this to export ENV vars with sensitive information such as secrets to become available on newly opened shells. Files under transient folder are git ignored by default to prevent from committing to a source control.

Shell Folder

A folder containing run command files (*rc) such as zshrc and bashrc for example. They usually contain custom content such as terminal colours, plugins etc.. which you'll probably want to backup and restore once needed.

To add such file as described:

Place it within <repo-root>/dotfiles/shell/
Run dotfiles sync shell

Note: Only the relevant shell *.rc file is being symlinked to $HOME folder, depends on the shell you are using. Currently supported shells are: bash, zsh.

macOS/Linux Settings Override

These are usually settings you have changed from the macOS/Linux GUI (Graphical User Interface) or from the terminal. In case you do have custom macOS/Linux settings, you'll probably want to back them via GitHub repository to be able to restore on-demand.

Restoring these settings could take place either on new machine, after OS re-install or in case you have tweaked some knobs and forgot how to revert.

To override macOS/Linux settings and preferences:

Find the terminal command corresponding to the setting you wish to backup
Add the command to a script files under:
- macOS - <repo-root>/dotfiles/os/mac
- Linux - <repo-root>/dotfiles/os/linux
Run dotfiles os <mac/linux> to override settings and preferences

Note: Check out macOS settings examples in here.

Demo

Click here for a running demo.

Summary

I hope this post helped to shed some light on how dotfiles could be useful for your local environment and you have found the way I'm automating dotfiles with Homebrew & macOS overrides insightful and helpful.

Please leave your comment, suggestion or any other input you think is relevant to this post in the discussion below.

Like this post?
You can find more by:

Checking out my blog: https://blog.zachinachshon.com
Following me on twitter: @zachinachshon

Thanks for reading! ❤️