DEV Community: Itzik Kotler

Useful faas-cli One-Liners

Itzik Kotler — Fri, 10 Jul 2020 13:20:21 +0000

OpenFaaS is a framework for building serverless functions with Docker and Kubernetes. Their goal, in their own words, is to make it easy for developers to deploy event-driven functions and microservices to Kubernetes without repetitive, boiler-plate coding.

faas-cli is the official CLI for OpenFaaS. In other words it is a command-line tool to help manage, prepare, and invoke functions.

In this post we’re going to cover 3 tips on how to use faas-cli to improve (and automate) our OpenFaaS workflows

1. Handle Multiple Functions from Multiple Files

When you start working with faas-cli it’s tempting to do this a lot:

faas-cli new --lang= ...

This will result in creating a new function (based on a template) but will also put it in it’s own YAML file. Now if we would like to leverage the faas-cli up command line option, we will need to find a way to run it on multiple YAML files. Here’s a one liner that will do it:

ls *.yml | awk '{ print $1 }' | xargs -I {} faas-cli up -f {}

This will build, push and deploy all of our functions in all of our YAML files. If we would like to remove all our functions from all our YAML files, just change faas-cli up to faas-cli remove -f:

ls *.yml | awk '{ print $1 }' | xargs -I {} faas-cli remove -f {}

Another way to remove all the functions (if you don’t have the YAML files) is this:

faas-cli list | tail -n +2 | awk '{ print $1 }' | xargs -I {} faas-cli remove {}

This will enumerate the output of faas-cli list instead of operating on the functions in the YAML files

2. Merge Multiple Functions from Multiple Files to a Single YAML Stack File

Consolidating all your functions to one YAML file has many benefits and you can easily do it using the yq utility:

yq merge *.yml > stack.yml

This will simplify your workflow, reduce your need for one liners and … it has more benefits! So keep reading the next tip :-)

3. Use Multiple Cores to Build & Deploy your Single YAML Stack File

The faas-cli lets you parallelize the build, push and deploy process by specifying the --parallel command line option. The --parallel takes an integer that will define how many concurrent build actions should be performed. A good starting point will be to pass nproc output to it:

faas-cli up -f stack.yml --parallel `nproc`

The output of the nproc utility is the number of processing units available on the computer that you are running this command on. So ideally you can maximize your CPUs in the process.

And of course, removing is now easy as:

faas-cli remove -f stack.yml

What’s next?

Go ahead and apply some of the tips above to your current OpenFaaS projects and workflows. If you are looking for more tips, check out this post by Alex Ellis. Do you have a favorite faas-cli trick? Feel free to leave it in your comments below.

Big thanks to Alex Ellis for reviewing this post

Running OpenFaaS and MongoDB on Raspbian 64bit

Itzik Kotler — Tue, 09 Jun 2020 13:17:54 +0000

In this tutorial we’ll get a Lightweight kubernetes running on Raspberry Pi 4B/3B/3B+ as ARM64 and set up OpenFaaS with MongoDB.

Raspberry Pi OS (64 bit) vs Raspbian 64bit:

To avoid confusion, this article will NOT talk about Raspberry Pi 64bit OS that is currently in beta. Instead, we will focus on Raspbian which is a 32bit but is capable of ALSO running 64bit applications. I assume there are plenty of people who still run Raspbian and will keep running in the foreseeable future.

We’ll use the following software:

raspbian-nspawn-64 package
k3sup tool
arkade tool
OpenFaaS Platform
MongoDB

And my hardware setup includes: A cluster of 10 x Raspberry Pi 4B with 4GB each.

Step #1: Enabling ARM64 & 64bit Userland (aka. 64bit Shell):

The default OS of Raspberry Pi (aka. Raspbian) as of now is completely 32bit, this is often abbreviated as armv7l or armhf. However, Raspberry Pi’s CPU starting from model 3B supports 64bit. In this step we will enable the 64bit kernel and setup a 64bit shell.

Thanks to the great work of sakaki's this is as easy as:

sudo apt-get update && sudo apt-get install -y raspbian-nspawn-64

You can read more about it here:

https://github.com/sakaki-/raspbian-nspawn-64.

Make sure you select ‘Yes’ in the dialog and reboot your Pi before proceeding to the next step.

Step #2: Using Docker/Containers Inside The 64bit Userland:

Since we’re use containers, we need to make some adjustments in our 64bit Userland:

In the Wiki page of the raspbian-nspawn-64 project there’s an article with a tutorial that explains how to get there:

https://github.com/sakaki-/raspbian-nspawn-64/wiki/Using-Docker-Inside-the-Container

Please follow all the instructions until the ‘Installing Docker in the 64-bit Debian Buster Container’ chapter. You can go ahead and install Docker if you want, that’s optional and can be handy but it won't be used by k3s.

Step 3#: Installing k3s (i.e., Lightweight Kubernetes) using k3sup and OpenFaaS using arakde:

Instead of getting Kubernetes (i.e., k8s) installed on our IoT edge, we’ll use k3s. k3s is a lightweight Kubernetes distribution, which is easy to install. To automate and simplify our workflow we’ll use k3sup and arkade.

Alex Ellis who wrote those great tools also wrote an excellent tutorial on how to use ‘em to bootstrap your Kubernetes (with OpenFaaS):

https://medium.com/@alexellisuk/walk-through-install-kubernetes-to-your-raspberry-pi-in-15-minutes-84a8492dc95a

To avoid confusion, in our current situation you should start from the ‘Get your CLI tools’ chapter. DO NOT flash the OS image otherwise you would lose all the progress from the previous steps.

At the end of this you should have a working k3s cluster with OpenFaaS deployed and you can verify it’s indeed ARM64 by running the following command:

kubectl describe nodes

Or in short:

kubectl describe nodes | grep "Architecture"

And confirm it’s indeed:

Architecture: arm64

Step #4: Installing MongoDB:

Unfortunately arkade doesn't support installing MongoDB on ARM64, and as of now the helm charts are also not supporting ARM64. This means we’ll have to do some hacking.

We’re going to use MongoDB from Docker’s Official Images library:

https://hub.docker.com/_/mongo

That supports ARM64. We’ll base our deployment on bitnami's helm chart but make the necessary modifications as the container image is different. Copy and paste below into a mongodb.yaml file:

# Source: mongodb/templates/svc-standalone.yaml
apiVersion: v1
kind: Service
metadata:
  name: db-mongodb
  namespace: default
  labels:
    app: mongodb
    release: "db"
spec:
  type: ClusterIP
  ports:
    - name: mongodb
      port: 27017
      targetPort: mongodb
  selector:
    app: mongodb
    release: "db"
---
# Source: mongodb/templates/deployment-standalone.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: db-mongodb
  namespace: default
  labels:
    app: mongodb
    release: "db"
spec:
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: mongodb
      release: "db"
  template:
    metadata:
      labels:
        app: mongodb
        release: "db"
    spec:
      containers:
        - name: db-mongodb
          image: docker.io/library/mongo:latest
          imagePullPolicy: "IfNotPresent"
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              value: "root"
            - name: MONGO_INITDB_ROOT_PASSWORD
              value: "example"
            - name: MONGODB_ENABLE_IPV6
              value: "no"
            - name: MONGODB_ENABLE_DIRECTORY_PER_DB
              value: "no"
          ports:
            - name: mongodb
              containerPort: 27017
          livenessProbe:
            exec:
              command:
                - mongo
                - --eval
                - "db.adminCommand('ping')"
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          readinessProbe:
            exec:
              command:
                - mongo
                - --eval
                - "db.adminCommand('ping')"
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          volumeMounts:
            - name: data
              mountPath: /data/db
              subPath:
          resources:
            {}
      volumes:
        - name: data
          emptyDir: {}

The mongodb.yaml above is not designed for production deployment. It doesn’t include persistent storage settings and has a hardcoded login and password to name a few caveats. However, it’s good enough for our proof-of-concept.

Deploying it is as easy as:

kubectl apply -f mongodb.yaml

And testing it (i.e., connecting to to MongoDB) is as easy as:

kubectl run --namespace default db-mongodb-client --rm --tty -i --restart='Never' --image docker.io/library/mongo:latest --command -- mongo admin --host db-mongodb --authenticationDatabase admin -u root -p example

The latter adopted from the NOTES.txt of bitnami’s MongoDB helm chart.

End-to-end Project:

We have taken all the steps above in order to finally be able to have storage for our OpenFaaS functions. Alex wrote an awesome tutorial on this topic here:

https://www.openfaas.com/blog/get-started-with-python-mongo/

Again, to avoid confusion, you should follow the tutorial from the ‘Create the hello-python3 function’ chapter.

Conclusions:

In this tutorial we’ve taken multiple steps to upgrade our cluster from 32bit to 64bit, quickly setup k3s with OpenFaaS, hack a MongoDB helm chart and get to a situation where we’ve got storage for our functions.

Big thanks to Alex Ellis for reviewing & contributing to this article!

Making HPA more responsive for resource-based scaling in OpenFaaS

Itzik Kotler — Fri, 15 May 2020 18:17:08 +0000

I've got a Raspberry Pi 4 cluster that is running Kubernetes. Thanks to k3sup and arkade I got it up and running in jiffy. Here are a few tutorials that I've followed to get there:

Having said that k3s is a little bit different and this post is about this difference when it comes to how to change the Kubernetes Horizontal Pod Autoscaling "cool down" period. (aka. --horizontal-pod-autoscaler-downscale-stabilization) in it.

Recently I've started playing with HPAv2 (aka. Kubernetes Horizontal Pod Autoscaling) and OpenFaaS. The reason for that is that my OpenFaaS functions tend to be CPU and MEM intensive (as oppose to high API hit rates, which is where the built-in OpenFaaS autoscaler/alertmanager is focused).

OpenFaaS and HPAv2 play nicely together and I've started by followed the guide here:

https://docs.openfaas.com/tutorials/kubernetes-hpa/

However, as correctly stated in the document the HPAv2 scale-down is a slow process (i.e., default of 5 minutes). The reason for that is:

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

And to be more specific:

When managing the scale of a group of replicas using the Horizontal Pod Autoscaler, it is possible that the number of replicas keeps fluctuating frequently due to the dynamic nature of the metrics evaluated. This is sometimes referred to as thrashing.

Starting from v1.6, a cluster operator can mitigate this problem by tuning the global HPA settings exposed as flags for the kube-controller-manager component:

Starting from v1.12, a new algorithmic update removes the need for the upscale delay.

--horizontal-pod-autoscaler-downscale-stabilization: The value for this option is a duration that specifies how long the autoscaler has to wait before another downscale operation can be performed after the current one has completed. The default value is 5 minutes (5m0s).

Now I'm using k3s (v1.17.2+k3s1) and in order to change it I need to pass a new value to kube-controller-manager. k3s is awesome, but it doesn't include the default path of:

/etc/kubernetes/manifests/kube-controller-manager.yaml on the master node

So I did some poking and found that it's possible to pass this flag/value to k3s upon start. In other words: We need to shutdown our k3s master node, edit the systemd unit of k3s, add this flag in runtime, and start it again.

Here are the steps:

SSH to your master node
Run: sudo systemctl stop k3s.service
Edit /etc/systemd/system/k3s.service with your favorite editor (e.g., sudo vim /etc/systemd/system/k3s.service)
Go to ExecStart section and append:

'--kube-controller-manager-arg' \
'horizontal-pod-autoscaler-downscale-stabilization=1m'

To command line options. Here's an example of my ExecStart Entry:

ExecStart=/usr/local/bin/k3s \
    server \
    '--tls-san' \
    '192.168.86.180' \
    '--no-deploy' \
    'servicelb' \
    '--no-deploy' \
    'traefik' \
    '--kube-controller-manager-arg' \
    'horizontal-pod-autoscaler-downscale-stabilization=1m'

(Note in my setup I've also disabled the built-in LB and traefik v1 ingress; this was done for me when I've passed --no-extras to k3sup)

Run: sudo systemctl daemon-reload
Run: sudo systemctl start k3s.service

That's it!

We have successfully changed the "cool down" period to one minute (i.e., 1m) in k3s. Now it's time to go back to our OpenFaaS functions and enjoy a more responsive resource-based scaling