DEV Community: Martin Heinz

Advanced Features of Kubernetes' Horizontal Pod Autoscaler

Martin Heinz — Mon, 04 Jul 2022 19:56:31 +0000

Most people who use Kubernetes know that you can scale applications using Horizontal Pod Autoscaler (HPA) based on their CPU or memory usage. There are however many more features of HPA that you can use to customize scaling behaviour of your application, such as scaling using custom application metrics or external metrics, as well as alpha/beta features like "scaling to zero" or container metrics scaling.

So, in this article we will explore all of these options so that we can take full advantage of all available features of HPA and to get a head start on the features that are coming in future Kubernetes releases.

Setup

Before we get started with scaling, we first need a testing environment. For that we will use KinD (Kubernetes in Docker) cluster defined by the following YAML:

# cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  HPAScaleToZero: true
  HPAContainerMetrics: true
  LogarithmicScaleDown: true
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker

This manifest configures the KinD cluster with 1 control plane node and 3 workers, additionally it enables a couple of feature gates related to autoscaling. These feature gates will later allow us to use some alpha/beta features of HPA. To create a cluster with the above configuration, you can run:

kind create cluster --config ./cluster.yaml --name autoscaling --image=kindest/node:v1.23.6

Apart from the cluster, we will also need an application that we will scale. For that we will use resource consumer tool and it's image, which are used in Kubernetes end-to-end testing. To deploy it, you can run:

kubectl create deployment resource-consumer --image=gcr.io/k8s-staging-e2e-test-images/resource-consumer:1.11
kubectl set resources deployment resource-consumer --requests=cpu=500m,memory=256Mi

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    app: resource-consumer
  name: resource-consumer
  namespace: default
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: resource-consumer
EOF

This application is very handy in this situation, as it allows us to simulate CPU and memory consumption of a Pod. It can also expose custom metrics which are needed for scaling based on custom/external metrics. To test this out we can run:

# Consume CPU (300m for 10min):
kubectl run curl --image=curlimages/curl:7.83.1 \
  --rm -it --restart=Never -- \
  curl --data "millicores=300&durationSec=600" http://resource-consumer:8080/ConsumeCPU

# Expose metric "custom_metric" with value 100 for 10min at endpoint /metrics
kubectl run curl --image=curlimages/curl:7.83.1 --rm \
  -it --restart=Never -- \
  curl --data "metric=custom_metric&delta=100&durationSec=600" http://resource-consumer:8080/BumpMetric

Next, we will also need to deploy services that collect metrics based on which we will later scale our test application. First of these is Kubernetes metrics-server which is usually available in cluster by default, but that's not the case in KinD, so to deploy it we need to run:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.0/components.yaml
kubectl patch -n kube-system deployment metrics-server --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

metrics-server allows us to monitor for basic metrics such as CPU and memory usage, but we also want to implement scaling based on custom metrics, such as the ones exposed by an application on its /metrics endpoint, or even external ones like queue depth of a queue running outside of cluster. For these we will need:

Prometheus Operator to gather the custom/external metrics.
ServiceMonitor object(s) to tell Prometheus how to scrape our application's metrics.
Prometheus adapter to get custom/external metrics from Prometheus instance into Kubernetes API.

You can refer to the end-to-end walkthrough for more details of the setup.

The above requires a lot of setup, so for purpose of this article and for your convenience, I've made a script and a set manifests that you can use to spin up KinD cluster along with all the required components. All you need to do is run setup.sh script from this repository.

After running the script, we can verify that everything is ready using following commands:

# To verify availability of metrics run:
kubectl top nodes
# NAME                        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
# autoscaling-control-plane   113m         0%     1024Mi          1%        
# autoscaling-worker          49m          0%     385Mi           0%        
# autoscaling-worker2         42m          0%     381Mi           0%        
# autoscaling-worker3         37m          0%     276Mi           0%

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq .  # also works with "pods"
# ...
#     {
#      "metadata": {
#        "name": "autoscaling-worker3",
#        "labels": { ... }
#      },
#      "window": "20s",
#      "usage": {
#        "cpu": "43077193n",
#        "memory": "283212Ki"
#      }

# To query/verify custom metrics:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . # also works for "external" instead of "custom"
# ...
# "name": "pods/custom_metric",
# "singularName": "",
# "namespaced": true,
# "kind": "MetricValueList",
# "verbs": [ "get" ]
# ...

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/custom_metric" | jq .
# ...
# {
#   "kind": "MetricValueList",
#   "apiVersion": "custom.metrics.k8s.io/v1beta1",
#   "metadata": {"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/custom_metric"},
#   "items": [{
#       "describedObject": {
#         "kind": "Pod",
#         "namespace": "default",
#         "name": "resource-consumer-6bf5898d6f-gzzgm",
#         "apiVersion": "/v1"
#       },
#       "metricName": "custom_metric", "value": "100",
#     }]}

More helpful commands can be found in output of above mentioned script or in the repository README.

Basic Autoscaling

Now that we have our infrastructure up-and-running, we can start scaling the test application. The simplest way to do so is to create HPA using command like kubectl autoscale deploy resource-consumer --min=1 --max=5 --cpu-percent=75, this however creates HPA with apiVersion of autoscaling/v1, which lacks most of the features.

So, instead, we will create the HPA with YAML, specifying autoscaling/v2 as a apiVersion:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: resource-consumer-v2
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: resource-consumer
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 75
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageValue: 200Mi

The above HPA will use basic metrics gathered from application Pod(s) by metrics-server. To test out the scaling we can simulate heavy memory usage:

kubectl run curl --image=curlimages/curl:7.83.1 \
  --rm -it --restart=Never -- \
  curl --data "megabytes=500&durationSec=600" http://resource-consumer:8080/ConsumeMem
kubectl get hpa -w
# NAME                   REFERENCE                      TARGETS                       MINPODS   MAXPODS   REPLICAS   AGE
# resource-consumer-v2   Deployment/resource-consumer   4689920/200Mi, 0%/75%         1         5         1          81s
# resource-consumer-v2   Deployment/resource-consumer   530415616/200Mi, 0%/75%       1         5         1          2m23s
# resource-consumer-v2   Deployment/resource-consumer   265820160/200Mi, 0%/75%       1         5         3          2m31s
# resource-consumer-v2   Deployment/resource-consumer   212226867200m/200Mi, 0%/75%   1         5         5          5m50s

Custom Metrics

Scaling based on CPU and memory usage is often enough, but we're after the advanced scaling options. First of them is scaling using custom metrics exposed by an application:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: resource-consumer-v2-custom
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: resource-consumer
  minReplicas: 1
  maxReplicas: 5
  metrics:
  # kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/custom_metric" | jq .
  - type: Pods
    pods:
      metric:
        name: custom_metric
      target:
        type: AverageValue
        averageValue: 100

This HPA is configured to scale the application based on the value of custom_metric that was scraped by Prometheus from application's /metrics endpoint. This will scale the application up if average value of specified metric across all pods (.target.type: AverageValue) goes over 100.

The above uses Pod metric to scale, but it's possible to specify any other object which has a metric attached to itself:

# ...
  - type: Object
    object:
      metric:
        name: custom_metric
      describedObject:
        apiVersion: v1
        kind: Service
        name: resource-consumer
      target:
        type: Value
        value: 100

This snippet achieves the same as the previous one, this time however, using Service instead of Pod as the source of the metric. It also shows that you can use direct comparison to measure the scaling threshold by setting .target.type to Value instead of AverageValue.

To figure out which objects expose metrics that you can use in scaling, you can traverse the API using kubectl get --raw. For example to look up the custom_metric for either Pod or Service you can use:

# Pod
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/custom_metric" | jq .
# Service
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/resource-consumer/custom_metric" | jq .
# Everything
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/" | jq .

Also, to help you troubleshoot, the HPA object provides a status stanza, that shows whether the applied metric was recognized:

kubectl get hpa resource-consumer-v2-custom -o json | jq .status.conditions
[
...
  {
    "lastTransitionTime": "2022-05-17T12:36:03Z",
    "message": "the HPA was able to successfully calculate a replica count from pods metric custom_metric",
    "reason": "ValidMetricFound",
    "status": "True",
    "type": "ScalingActive"
  },
...
]

Finally, to test out the behavior of the above HPA, we can bump the metric exposed by the application and see how the application scales up:

# Raise custom_metric to 150
kubectl run curl --image=curlimages/curl:7.83.1 \
  --rm -it --restart=Never -- curl \
  --data "metric=custom_metric&delta=150&durationSec=600" http://resource-consumer:8080/BumpMetric

kubectl get hpa -w
# NAME                          REFERENCE                      TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
# resource-consumer-v2-custom   Deployment/resource-consumer     0/100   1         5         1          10s
# resource-consumer-v2-custom   Deployment/resource-consumer   150/100   1         5         1          24s
# resource-consumer-v2-custom   Deployment/resource-consumer   150/100   1         5         2          40s
# resource-consumer-v2-custom   Deployment/resource-consumer   150/100   1         5         5          75s

External Metrics

To show full potential of HPA, we will also try scaling an application based on external metric. This would require us to scrape metrics from external system running outside of a cluster, such Kafka or PostgreSQL. We don't have that available, so instead we've configured Prometheus Adapter to treat certain metrics as external. The configuration that does this can be found here. All you need to know though is that with this test cluster, any application metrics prefixed with external will go to external metrics API. To test this out, we bump up such a metric and check if the API gets populated:

# Set external_queue_messages_ready to 150 for 10min
kubectl run curl --image=curlimages/curl:7.83.1 \
  --rm -it --restart=Never -- \
  curl --data "metric=external_queue_messages_ready&delta=150&durationSec=600" \
  http://resource-consumer:8080/BumpMetric

kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/external_queue_messages_ready | jq .
{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "items": [
    {
      "metricName": "external_queue_messages_ready",
      "value": "150"
    }
  ]
}

To then scale our deployment based on this metric we can use following HPA:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: resource-consumer-v2-external
spec:
# ...
  metrics:
  - type: External
    external:
      metric:
        name: external_queue_messages_ready
      target:
        type: Value
        value: 100

HPAScaleToZero

Now that we've gone through all the well known features of HPA, let's also take a look at the alpha/beta ones that we enabled using feature gates. First one being HPAScaleToZero.

As the name suggests, this will allow you to set minReplicas in HPA to zero, effectively turning the service off if there's no traffic. This can be useful in "bursty" workflow, for example in case where your application receives data from an external queue. In this use case the application can be safely scaled to zero when there are messages waiting to be processed.

With the feature gate enabled we can simply run:

kubectl patch hpa resource-consumer-v2-external -p '{"spec":{"minReplicas": 0}}'

Which sets the minimum replicas of previously shown HPA to zero.

Be aware though, that this will only work for metrics of type External or Object.

HPAContainerMetrics

Another feature gate that we can make use of is HPAContainerMetrics which allows us to use metrics of type: ContainerResource:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: resource-consumer-v2-container
spec:
# ...
  metrics:
  - type: ContainerResource
    containerResource:
      name: cpu
      container: resource-consumer
      target:
        type: Utilization
        averageUtilization: 75

This makes it possible to scale based on resource utilization of individual containers rather than whole Pod. This can be useful if you have multi-container Pod with application container and sidecar, and you want to ignore the sidecar and scale the deployment only based on the application container.

You can also view the breakdown of Pod/container metrics by running the following command:

POD=$(kubectl get pod -l app=resource-consumer -o jsonpath="{.items[0].metadata.name}")
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/$POD" | jq .

{
  "kind": "PodMetrics",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "name": "resource-consumer-6bf5898d6f-gzzgm",
    "namespace": "default",
  },
  "window": "16s",
  "containers": [{
      "name": "resource-consumer",
      "usage": {
        "cpu": "0",
        "memory": "11028Ki"
      }}]}

LogarithmicScaleDown

Last but not least is LogarithmicScaleDown feature flag.

Without this feature, the Pod that's been running for least amount of time gets deleted first during downscaling. That's not always ideal though as it can create imbalance in replica distribution because newer Pods tend serve less traffic than the older ones.

With this feature flag enabled, a semi-random selection of Pods will be used instead when selecting Pod to be deleted.

For a full rationale and algorithm details see KEP-2189.

Closing Thoughts

In this article, I tried to cover most of the things you can do with Kubernetes HPA to scale your application. There are however, many more tools and options for scaling applications running in Kubernetes, such as vertical pod autoscaler which can help to keep Pod resource requests and limits up-to-date.

Another option would be predictive HPA by Digital Ocean, which will try to predict how many replicas a resource should and application have.

Finally, autoscaling doesn't end with Pods - next step after setting up Pod autoscaling is to also set up cluster autoscaling to avoid running out of available resources in you whole cluster.

Data and System Visualization Tools That Will Boost Your Productivity

Martin Heinz — Mon, 13 Jun 2022 18:50:45 +0000

As files, datasets and configurations grow, it gets increasingly difficult to navigate them. There are however many tools out there, that can help you to be more productive when dealing with large JSON and YAML files, complicated regular expressions, confusing SQL database relationships, complex development environments and many others.

JSON

JSON is one of those formats which is nice for computers, but not for humans. Even relatively small JSON object can be pretty difficult to read and traverse, but there's a tool that can help!

JSON Visio is a tool that generates graph diagrams from JSON objects. These diagrams are much easier to navigate than the textual format and to make it even more convenient, the tool also allows you to search the nodes. Additionally, the generated diagrams can also be downloaded as image.

You can use the web version at https://jsonvisio.com/editor or also run it locally as Docker container.

Regular Expressions

Regular expressions (RegEx) are notorious for being extremely unreadable. There are 2 tools I recommend to help make sense of complex RegExes - first one them being https://regex101.com/:

Which can help you build and test RegExes, as well as break them down and identify its individual parts.

The second one is https://regex-vis.com which generates a graph from a RegEx which is very helpful for understanding what the expression actually does:

YAML

YAML, the language that was meant to be readable, expect it oftentimes isn't. AS we all know, long YAML documents with many levels of indentations can be very hard to navigate and troubleshoot.

To avoid spending unreasonable amount of time trying to find that one wrong indent, I recommend you use schema validation and let your IDE do all the work. You can use validation schemas from https://schemastore.org/json or custom schemas such as these for Kubernetes to validate your files. These will work both with JetBrains products (e.g. Pycharm, IntelliJ) as well as VSCode (see this guide)

If you prefer to use vim as your editor, I recommend using custom formatting that can help you spot mistakes. My preferred configuration looks like so:

# Add this line to "~/.vimrc"
autocmd FileType yaml setlocal ai et cuc sw=2 ts=2

And the resulting formatting will look like:

On top of the above-mentioned tools, it's also a good idea to use YAML linter such this one or its CLI equivalent, which will validate and cleanup your documents.

Finally, if you're working with OpenAPI/Swagger YAML specs, then you can use https://editor.swagger.io/ to view/validate/edit you schemas.

SQL

There's a lot of software for working with relational databases, most of them however focus on connecting to database instances and running SQL queries. These capabilities are very handy, but don't help with the fact that navigating database with possibly hundreds of tables can be very difficult. One tool that can solve that is Jailer:

Jailer is a tool which can - among other things - navigate through the database by following foreign keys. Couple videos showcasing all the features of Jailer can be found here.

Git

Git - a software with absolutely terrible user experience which we all use every day - could also use some tooling for navigating history (log), staging/committing files, viewing diffs or for example rebasing branches.

My tool of choice for all the above is IntelliJ/PyCharm git integration - in my opinion there's really no better tool for dealing with git-related stuff.

If you're not an IntelliJ/PyCharm user or if you prefer to stay in terminal, then following commands can make your life a little bit easier:

git log --graph --abbrev-commit --decorate --all \
    --format=format:"%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(dim white) \
    - %an%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n %C(white)%s%C(reset)"

The above command will print a somewhat readable and pretty graph of commits history, which would look something like:

If you also desire a bit improved diffing functionality, then you can use git diff ... --word-diff --color-words:

As you can see from above 2 examples, with enough effort you might be able to make commandline git experience somewhat bearable, for more tips like these see my previous article at https://martinheinz.dev/blog/43.

If you want to avoid bare-bones git CLI altogether, you can try using forgit - a TUI for using git interactively:

The above screenshot shows interactive and pretty git log. The tool supports pretty much all of the git subcommands, most importantly rebase, cherry-pick, diff and commit. For full list of features and more screenshots see project's repository.

Beyond using git or your IDEs features you can also grab other tools that can help with complex diffing of files. One very powerful tool for this is Difftastic. Difftastic language-aware diffing tools which has support for wide range of languages. For demo of the tool see this video.

If you want a tool specifically for diffing structured languages like JSON, XML, HTML, YAML, then Graphtage is a great choice. Graphtage compares documents semantically and will find differences correctly even when you change ordering of elements.

Finally, if you're more into visual tools, then you might find Meld useful as it provides similar experience to JetBrains' products.

Docker

On the DevOps side of things, when working with Docker, it's not uncommon to spin up bunch of containers with docker-compose and end-up with a mess that's very hard to troubleshoot.

Lazydocker is a great tool for dealing with multiple Docker containers at the same time. If you don't believe me, then check out the "elevator pitch" in the project's repository.

If you're more into browser-based tools you might want to try out Portainer which provides dashboard for navigating/inspecting Docker containers, volumes, images, etc.

Kubernetes

A lot of things to cover when it comes to Kubernetes, considering that each API resource could use a visualization tool. There are however, a couple notable areas (pain points really), that oftentimes require visual tool to setup/manage effectively.

First one being NetworkPolicy which can be visualized and also configured using https://editor.cilium.io/. Even if you prefer to craft your policies by hand, I would still recommend visually verifying them with this tool.

Another similar tool is Network Policy Viewer, which focuses just on the visualization of policies and has simpler and more readable diagrams.

I'd recommend using this collection of network policy recipes to test out these 2 tools and see how they can be helpful to your workflow.

Another pain point in configuring Kubernetes is RBAC, more specifically Roles, RoleBindings and their Cluster-scoped alternatives. There are a couple of tools that can help with those:

Krane is a tool that can generate graph showing relationships between all roles and subjects. Krane also has many more features, including RBAC risk assessment, reporting and alerting, as well as querying/interrogating RBAC rules with CypherQL.

A simpler alternative to Krane is rbac-tool, which can be installed as kubectl plugin. It can also analyze, audit, interrogate RBAC rules, but most importantly, it can visualize them:

Finally, if you're more concerned with easily configuring RBAC rather than viewing pretty diagrams, then Permission manager is a tool for you. See GIFs in GitHub repository for demonstration of what the tool can do.

Instead of specialized tools for things like network policies or RBAC you can also make use of general purpose dashboards, such as:

Lens - the Kubernetes IDE which brings some sanity into managing clusters, especially when paired with Lens Resource Map which displays Kubernetes resources and their relations as a real-time force-directed graph.

As usual, there's also CLI based tool which provides more capabilities than the bare kubectl. It's called k9s and definitely does make it easier to navigate, observe and manage your deployed applications in Kubernetes:

Closing Thoughts

This article focused on developer/DevOps tooling, but there are 2 more things - honorable mentions - that deserve a shout-out. First being Mermaid.js for creating beautiful diagrams (as a code), which is now integrated with GitHub markdown. The other one - MathJax - for visualizing mathematical expressions, as of recent also supported by GitHub markdown.

These are the tools I like or the tools I made a use of in the past, so it's by no means an exhaustive list of things out there. If you have found different (better?) tools, please do share, so others can make use of them too.

Stop Messing with Kubernetes Finalizers

Martin Heinz — Wed, 01 Jun 2022 20:25:50 +0000

We've all been there - it's frustrating seeing deletion of Kubernetes resource getting stuck, hang or take a very long time. You might have "solved" this using the terrible advice of removing finalizers or running kubectl delete ... --force --grace-period=0 to force immediate deletion. 99% of the time this is a horrible idea and in this article I will show you why.

Finalizers

Before we get into why force-deletion is a bad idea, we first need to talk about finalizers.

Finalizers are values in resource metadata that signal required pre-delete operations - they tell resource controller what operations need to be performed before object is deleted.

The most common one would be:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
  finalizers:
  - kubernetes.io/pvc-protection
...

Their purpose is to stop a resource from being deleted, while controller or Kubernetes Operator cleanly and gracefully cleans-up any dependant objects such as underlying storage devices.

When you delete an object which has a finalizer, deletionTimestamp is added to resource metadata making the object read-only. Only exception to the read-only rule, is that finalizers can be removed. Once all finalizers are gone, the object is queued to be deleted.

It's important to understand that finalizers are just items/keys in resource metadata. Finalizers don't specify the code to execute. They have to be added/removed by the resource controller.

Also, don't confuse finalizers with Owner References. .metadata.OwnerReferences field specify parent/child relations between objects such as Deployment -> ReplicaSet -> Pod. When you delete an object such as Deployment a whole tree of child objects can be deleted. This process (deletion) is automatic, unlike with finalizers, where controller needs to take some action and remove the finalizer field.

What Could Go Wrong?

As mentioned earlier, the most common finalizer you might encounter is the one attached to Persistent Volume (PV) or Persistent Volume Claim (PVC). This finalizer protects the storage from being deleted while it's in use by a Pod. Therefore, if the PV or PVC doesn't want to delete, it most likely means that it's still mounted by a Pod. If you decide force-delete PV, be aware that backing storage in Cloud or any other infrastructure might not get deleted, therefore you might leave a dangling resource, which still costs you money.

Another example is a Namespace which can get stuck in Terminating state because resources still exist in the namespace that the namespace controller is unable to remove. Forcing deletion of namespace can leave dangling resources in your cluster which include for example Cloud provider's load balancer which might be very hard to track down later.

While not necessarily related to finalizers, it's good to mention that resources can get stuck for many other reasons other than waiting for finalizers:

The simplest example would be Pod being stuck in Terminating state, which usually signals issue with Node on which the Pod runs. "Solving" this with kubectl delete pod --grace-period=0 --force ... will remove the Pod from API server (etcd), but it might still be running on the Node, which is definitely not desirable.

Another example would be a StatefulSet, where Pod force-deletion can create problems because Pods have fixed identities (pod-0,pod-1). A distributed system might depend on these names/identities - if the Pod is force-deleted, but still runs on the node, you can end-up with 2 pods with same identity when StatefulSet controller replaces the original "deleted" Pod. These 2 Pods might then attempt to access same storage, which can lead to corrupted data. More on this in docs.

Finalizers in The Wild

We now know that we shouldn't mess with resources that have finalizers attacked to them, but which resources are these?

The 3 most common ones you will encounter in "vanilla" Kubernetes are kubernetes.io/pv-protection and kubernetes.io/pvc-protection related to Persistent Volumes and Persistent Volume Claims respectively (plus couple more introduced in v1.23) as well as kubernetes finalizer present on Namespaces. The last one however isn't in .metadata.finalizers field but rather in .spec.finalizers - this special case is described in architecture document.

Besides these "vanilla" finalizers, you might encounter many more if you install Kubernetes Operators which often perform pre-deletion logic on their custom resources. A quick search through code of some popular projects turn up the following:

Istio - istio-finalizer.install.istio.io
Cert Manager - finalizer.acme.cert-manager.io
Strimzi (Kafka) - service.kubernetes.io/load-balancer-cleanup
Quay - quay-operator/finalizer
Ceph/Rook - ceph.rook.io/disaster-protection
ArgoCD - argoproj.io/finalizer
Litmus Chaos - chaosengine.litmuschaos.io/finalizer

If you want to find all the finalizers that are present in your cluster, then you will have to run the following command against each resource type:

kubectl get some-resource -o custom-columns=Kind:.kind,Name:.metadata.name,Finalizers:.metadata.finalizers

You can use kubectl api-resources to get a list resources types available in your cluster.

Regardless of which finalizer is stopping deletion of your resources, the negative effects of force-deleting those resources will be generally the same, which is something being left behind, be it storage, load balancer, or a simple pod.

Also, the proper solution will be generally the same, which is finding the finalizer that's stopping the deletion, figuring out what is its purpose - possibly by looking at the source code of the controller/operator - and resolving whatever is blocking the controller from removing the finalizer.

If you decide to force-delete the problematic resources anyway, then the solution would be:

kubectl patch some-resource/some-name \
    --type json \
    --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'

One exception would be Namespace, which has finalize API method which is usually called when all resources in said Namespace are cleaned-up. If the Namespace refuses to delete even when there are no resources left to delete, then you can call the method yourself:

cat <<EOF | curl -X PUT \
  localhost:12345/api/v1/namespaces/my-namespace/finalize \
  -H "Content-Type: application/json" \
  --data-binary @-
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "my-namespace"
  },
  "spec": {
    "finalizers": null,
  }
}
EOF

Building Your Own

Now that we know what they are and how they work, it should be clear that they're quite useful, so let's see how we can apply them to our own resources and workloads.

Kubernetes ecosystem is based around Go, but for simplicity's sake I will use Python here. If you're not familiar with Python Kubernetes client library, consider reading my previous article first - Automate All the Boring Kubernetes Operations with Python.

Before we start using finalizers, we first need to create some resource in a cluster - in this case a Deployment:

# initialize the client library...

deployment_name = "my-deploy"
ns = "default"

v1 = client.AppsV1Api(api_client)

deployment_manifest = client.V1Deployment(
    api_version="apps/v1",
    kind="Deployment",
    metadata=client.V1ObjectMeta(name=deployment_name),
    spec=client.V1DeploymentSpec(
        replicas=3,
        selector=client.V1LabelSelector(match_labels={
            "app": "nginx"
        }),
        template=client.V1PodTemplateSpec(
            metadata=client.V1ObjectMeta(labels={"app": "nginx"}),
            spec=client.V1PodSpec(
                containers=[client.V1Container(name="nginx",
                                               image="nginx:1.21.6",
                                               ports=[client.V1ContainerPort(container_port=80)]
                                               )]))))

response = v1.create_namespaced_deployment(body=deployment_manifest, namespace=ns)

The above code creates a sample Deployment called my-deploy, at this time without any finalizer. To add a couple finalizers we will use following patch:

finalizers = ["test/finalizer1", "test/finalizer2"]

v1.patch_namespaced_deployment(deployment_name, ns, {"metadata": {"finalizers": finalizers}})

while True:
    try:
        response = v1.read_namespaced_deployment_status(name=deployment_name, namespace=ns)
        if response.status.available_replicas != 3:
            print("Waiting for Deployment to become ready...")
            time.sleep(5)
        else:
            break
    except ApiException as e:
        print(f"Exception when calling AppsV1Api -> read_namespaced_deployment_status: {e}\n")

The important part here is a call to patch_namespaced_deployment which sets .metadata.finalizers to a list of finalizers we defined. Each of these must be fully qualified, meaning they must contain / as they need to adhere to DNS-1123 specification. Ideally, to make them more understandable you should use format like kubernetes.io/pvc-protection, where you prefix it with hostname of your service which is related to the controller responsible for the finalizer.

Rest of the code in the above snippet simply makes sure that the replicas of the Deployment are available after which we can proceed with managing the finalizers:

from kubernetes import client, watch

def finalize(deployment, namespace, finalizer):
    print(f"Do some pre-deletion task related to the {finalizer} present in {namespace}/{deployment}")
    ...

v1 = client.AppsV1Api(api_client)
w = watch.Watch()
for deploy in w.stream(partial(v1.list_namespaced_deployment, namespace=ns)):
    print(f"Deploy - Message: Event type: {deploy['type']}, Deployment {deploy['object']['metadata']['name']} was changed.")
    if deploy['type'] == "MODIFIED" and "deletionTimestamp" in deploy['object']['metadata']:

        fins = deploy['object']['metadata']['finalizers']
        f = fins[0]
        finalize(deploy['object']['metadata']['name'], ns, f)
        new_fins = list(set(fins) - {f})
        body = [{
            "kind": "Deployment",
            "apiVersion": "apps/v1",
            "metadata": {
                "name": deploy['object']['metadata']['name'],
            },
            "op": "replace",
            "path": f"/metadata/finalizers",
            "value": new_fins
        }]
        resp = v1.patch_namespaced_deployment(name=deploy['object']['metadata']['name'],
                                              namespace=ns,
                                              body=body,
                                              field_manager="json")
    elif deploy['type'] == "DELETED":
        print(f"{deploy['object']['metadata']['name']} successfully deleted.")
print("Finished namespace stream.")

The general sequence here is as follows:

We start by watching the desired resource - in this case a Deployment - for any changes/events. We then look for the events that relate to modifications to the resource and we specifically check whether deletionTimestamp is present. If it's there, we grab list of finalizers from resource's metadata and start processing first of them. We first perform all necessary pre-deletion tasks with finalize function, after which we apply patch to the resource with original list of finalizers minus the one we processed.

If the patch in Python looks complicated to you, then just know that it's an equivalent to the following kubectl command:

kubectl patch deployment/my-deploy \
  --type json \
  --patch='[ { "op": "replace", "path": "/metadata/finalizers", "value": [test/finalizer1] } ]'

If the patch is accepted, we will receive another modification event at which point we will process another finalizer. We repeat that until all finalizers are gone. At that point resource gets automatically deleted.

Be aware that you might receive the events more than once, therefore it's important to make the pre-deletion logic idempotent.

If you run the above code snippets and then execute kubectl delete deployment my-deploy, then you should see logs like:

# Finalizers added to Deployment
Deploy - Message: Event type: ADDED, Deployment my-deploy was changed.
# "kubectl delete" gets executed, "deletionTimestamp" is added
Deploy - Message: Event type: MODIFIED, Deployment my-deploy was changed.
# First finalizer is removed...
Do some pre-deletion task related to the test/finalizer1 present in default/my-deploy
# Another "MODIFIED" event comes in, Second finalizer is removed...
Deploy - Message: Event type: MODIFIED, Deployment my-deploy was changed.
Do some pre-deletion task related to the test/finalizer2 present in default/my-deploy
# Finalizers are gone "DELETED" event comes - Deployment is gone.
Deploy - Message: Event type: DELETED, Deployment my-deploy was changed.
my-deploy successfully deleted.

The above demonstration using Python works, but isn't exactly robust. In real-world scenario you'd most likely want to use Operator framework either through kopf in case of Python, or more usually with Kubebuilder for Go. Kubebuilder docs also includes whole page on how to use finalizers, including sample code.

If you don't want to implement whole Kubernetes Operator, you can also choose to build Mutating Webhook which is described in Dynamic Admission Control docs. The process there would be the same - receive the event, process your business logic and remove the finalizer.

Conclusion

One think you should take away from this article is that you might want to think twice before using --force --grace-period=0 or removing finalizers from resources. There might be situations when it's OK to ignore finalizer, but for your own sake, investigate before using the nuclear solution and be aware of possible consequences as doing so might hide a systemic problem in your cluster.

Automate All the Boring Kubernetes Operations with Python

Martin Heinz — Wed, 18 May 2022 10:58:24 +0000

Kubernetes became a de-facto standard in recent years and many of us - both DevOps engineers and developers alike - use it on daily basis. Many of the task that we perform are however, same, boring and easy to automate. Oftentimes it's simple enough to whip up a quick shell script with a bunch of kubectl commands, but for more complicated automation tasks bash just isn't good enough, and you need the power of proper language, such as Python.

So, in this article we will look at how you can leverage Kubernetes Python Client library to automate whatever annoying Kubernetes task you might be dealing with!

Playground

Before we start playing with the Kubernetes client, we first need to create a playground cluster where we can safely test things out. We will use KinD (Kubernetes in Docker), which you can install from here.

We will use the following cluster configuration:

# kind.yaml
# https://kind.sigs.k8s.io/docs/user/configuration/
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
name: api-playground
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker

To create cluster from above configuration, you can run:

kind create cluster --image kindest/node:v1.23.5 --config=kind.yaml

kubectl cluster-info --context kind-api-playground
# Kubernetes control plane is running at https://127.0.0.1:36599
# CoreDNS is running at https://127.0.0.1:36599/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

kubectl get nodes
# NAME                           STATUS     ROLES                  AGE   VERSION
# api-playground-control-plane   Ready      control-plane,master   58s   v1.23.5
# api-playground-worker          Ready      <none>                 27s   v1.23.5
# api-playground-worker2         NotReady   <none>                 27s   v1.23.5
# api-playground-worker3         NotReady   <none>                 27s   v1.23.5

With cluster up-and-running, we also need to install the client library (optionally, inside virtual environment):

python3 -m venv venv
source venv/bin/activate
pip install kubernetes

Authentication

To perform any action inside our Kubernetes cluster we first need to authenticate.

We will use long-lived tokens so that we don't need to go through the authentication flow repeatedly. Long-lived tokens can be created by creating a ServiceAccount:

kubectl create sa playground
kubectl describe sa playground

Name:                playground
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   playground-token-v8bq7
Tokens:              playground-token-v8bq7
Events:              <none>

export KIND_TOKEN=$(kubectl get secret playground-token-v8bq7 -o json | jq -r .data.token | base64 --decode)

Using a service account also has the benefit, that it's not tied to any single person, which is always preferable for automation purposes.

Token from the output above can be then used in requests:

curl -k -X GET -H "Authorization: Bearer $KIND_TOKEN" https://127.0.0.1:36599/apis

We're now authenticated, but not authorized to do much of anything. Therefore, next we need to create a Role and bind it to the ServiceAccount so that we can perform actions on resources:

kubectl create clusterrole manage-pods \
    --verb=get --verb=list --verb=watch --verb=create --verb=update --verb=patch --verb=delete \
    --resource=pods

kubectl -n default create rolebinding sa-manage-pods \
    --clusterrole=manage-pods \
    --serviceaccount=default:playground

The above gives our service account permission to perform any action on pods, limited to default namespace.

You should always keep your roles very narrow and specific, but playing around in KinD, it makes sense to apply cluster-wide admin role:

kubectl create clusterrolebinding sa-cluster-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=default:playground

Raw Requests

To get a better understanding of what is kubectl and also the client doing under the hood, we will start with raw HTTP requests using curl.

The easiest way to find out what requests are being made under the hood, is to run the desired kubectl command with -v 10 which will output complete curl commands:

kubectl get pods -v 10
# <snip>
curl -k -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json..." \
    'https://127.0.0.1:36599/api/v1/namespaces/default/pods?limit=500'
# <snip>

The output with loglevel 10 will be very verbose, but somewhere it there, you will find the above curl command.

Add a Bearer token header in the above curl command with your long-lived token and you should be able to perform same actions as kubectl, such as:

curl -s -k -XGET -H "Authorization: Bearer $KIND_TOKEN" -H "Accept: application/json, */*" -H "Content-Type: application/json" \
    -H "kubernetes/$Format" 'https://127.0.0.1:36599/api/v1/namespaces/default/pods/example' | jq .status.phase
# "Running"

In case there's request body needed, look up which fields need to be included in the request. For example when creating a Pod, we can use API described here, which results in following request:

curl -k -XPOST -H "Authorization: Bearer $KIND_TOKEN" -H "Accept: application/json, */*" -H "Content-Type: application/json" \
    -H "kubernetes/$Format" https://127.0.0.1:36599/api/v1/namespaces/default/pods -d@pod.json

# To confirm
kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
example   0/1     Running   0          7s

Refer to the Kubernetes API reference for object attributes. Additionally, you can also view OpenAPI definition with:

curl -k -X GET -H "Authorization: Bearer $KIND_TOKEN" https://127.0.0.1:36599/apis

Interacting with Kubernetes directly using REST API might be a bit clunky, but there are situations where it might make sense to use it. That includes interacting with APIs that have no equivalent kubectl command or for example in case you're using different distribution of Kubernetes - such as OpenShift - which exposes additional APIs not covered by either kubectl or client SDK.

Python Client

Moving onto the Python client itself now. We need to go through the same step as with kubectl or curl. First being, authentication:

from kubernetes import client
import os

configuration = client.Configuration()
configuration.api_key_prefix["authorization"] = "Bearer"
configuration.host = "https://127.0.0.1:36599"
configuration.api_key["authorization"] = os.getenv("KIND_TOKEN", None)
configuration.verify_ssl = False  # Only for testing with KinD!
api_client = client.ApiClient(configuration)
v1 = client.CoreV1Api(api_client)

ret = v1.list_namespaced_pod(namespace="default", watch=False)
for pod in ret.items:
    print(f"Name: {pod.metadata.name}, Namespace: {pod.metadata.namespace} IP: {pod.status.pod_ip}")
    # Name: example, Namespace: default IP: 10.244.2.2

First we define configuration object which tells the client that we will authenticate using Bearer token. Considering that our KinD cluster doesn't use SSL, we disable it, in real cluster however, you should never do that.

To test out the configuration, we use list_namespaced_pod method of API client to get all pods in the default namespace, and we print out their name, namespace and IP.

Now, for a more realistic task, let's create a Deployment:

deployment_name = "my-deploy"
deployment_manifest = {
    "apiVersion": "apps/v1",
    "kind": "Deployment",
    "metadata": {"name": deployment_name, "namespace": "default"},
    "spec": {"replicas": 3,
             "selector": {
                "matchLabels": {
                    "app": "nginx"
                }},
        "template": {"metadata": {"labels": {"app": "nginx"}},
            "spec": {"containers": [
                {"name": "nginx", "image": "nginx:1.21.6", "ports": [{"containerPort": 80}]}]
            }
        },
    }
}

import time
from kubernetes.client.rest import ApiException

v1 = client.AppsV1Api(api_client)

response = v1.create_namespaced_deployment(body=deployment_manifest, namespace="default")
while True:
    try:
        response = v1.read_namespaced_deployment_status(name=deployment_name, namespace="default")
        if response.status.available_replicas != 3:
            print("Waiting for Deployment to become ready...")
            time.sleep(5)
        else:
            break
    except ApiException as e:
        print(f"Exception when calling AppsV1Api -> read_namespaced_deployment_status: {e}\n")

In addition to creating the Deployment, we also wait for its pods to become available. We do that by querying Deployment status and checking number of available replicas.

Also, notice the pattern in function names, such as create_namespaced_deployment. To make it more obvious let's look at couple more:

replace_namespaced_cron_job
patch_namespaced_stateful_set
list_namespaced_horizontal_pod_autoscaler
read_namespaced_daemon_set
read_custom_resource_definition

All of these are in format operation_namespaced_resource or just operation_resource for global resources. They can be additionally suffixed with _status or _scale for methods that perform operations on resource status such as read_namespaced_deployment_status or resource scale such as patch_namespaced_stateful_set_scale.

Another thing to highlight is that in the above example we performed the actions using client.AppsV1Api which allows us to work with all the resources that belong to apiVersion: apps/v1. If we - for example - wanted to use CronJob we would instead choose BatchV1Api (which is apiVersion: batch/v1 in YAML format) or for PVCs we would choose CoreV1Api because of apiVersion: v1 - you get the gist.

As you can imagine, that's a lot of functions to choose from, luckily all of them are listed in docs and you can click on any one of them to get an example of its usage.

Beyond basic CRUD operations, it's also possible to continuously watch objects for changes. Obvious choice is to watch Events:

from kubernetes import client, watch

v1 = client.CoreV1Api(api_client)
count = 10
w = watch.Watch()
for event in w.stream(partial(v1.list_namespaced_event, namespace="default"), timeout_seconds=10):
    print(f"Event - Message: {event['object']['message']} at {event['object']['metadata']['creationTimestamp']}")
    count -= 1
    if not count:
        w.stop()
print("Finished namespace stream.")

# Event - Message: Successfully assigned default/my-deploy-cb69f686c-2dspd to api-playground-worker2 at 2022-04-19T11:18:25Z
# Event - Message: Container image "nginx:1.21.6" already present on machine at 2022-04-19T11:18:26Z
# Event - Message: Created container nginx at 2022-04-19T11:18:26Z
# Event - Message: Started container nginx at 2022-04-19T11:18:26Z

Here we chose to watch events in default namespace. We take the first 10 events and then close the stream. If we wanted to continuously monitor the resources we would just remove the timeout_seconds and the w.stop() call.

In the first example you saw that we used plain Python dict to define the Deployment object which we passed to the client. Alternatively though, we can use a more OOP style by using API Models (classes) provided by the library:

v1 = client.AppsV1Api(api_client)

deployment_manifest = client.V1Deployment(
    api_version="apps/v1",
    kind="Deployment",
    metadata=client.V1ObjectMeta(name=deployment_name),
    spec=client.V1DeploymentSpec(
        replicas=3,
        selector=client.V1LabelSelector(match_labels={
         "app": "nginx"
        }),
        template=client.V1PodTemplateSpec(
            metadata=client.V1ObjectMeta(labels={"app": "nginx"}),
            spec=client.V1PodSpec(
                containers=[client.V1Container(name="nginx",
                                               image="nginx:1.21.6",
                                               ports=[client.V1ContainerPort(container_port=80)]
                                               )]))
        )
)

response = v1.create_namespaced_deployment(body=deployment_manifest, namespace="default")

Trying to figure out which model you should use for each argument is a losing battle, tough. When creating resources like shown above, you should always use documentation for models and traverse the links as you create the individual sub-objects to figure out what values/types are expected in each field.

Handy Examples

You should now have a basic idea about how the client works, so let's take a look at some handy examples and snippets that might help you automate daily Kubernetes operations.

A very common thing you might want to perform is a Deployment rollout - usually done with kubectl rollout restart. There's however no API to do this. The way kubectl does it is by updating Deployment Annotations, more specifically, setting kubectl.kubernetes.io/restartedAt to current time. This works because any change made to Pod spec causes a restart.

If we want to perform a restart using Python client we need to do the same:

from kubernetes import dynamic
from kubernetes.client import api_client  # Careful - different import - not the same as previous client!
import datetime

client = dynamic.DynamicClient(api_client.ApiClient(configuration=configuration))

api = client.resources.get(api_version="apps/v1", kind="Deployment")

# Even though the Deployment manifest was previously created with class model, it still behaves as dictionary:
deployment_manifest["spec"]["template"]["metadata"]["annotations"] = {
    "kubectl.kubernetes.io/restartedAt": datetime.datetime.utcnow().isoformat()
}

deployment_patched = api.patch(body=deployment_manifest, name=deployment_name, namespace="default")

Another common operation is scaling a Deployment, this one fortunately has an API function we can use:

from kubernetes import client

api_client = client.ApiClient(configuration)
apps_v1 = client.AppsV1Api(api_client)

# The body can be of different patch types - https://github.com/kubernetes-client/python/issues/1206#issuecomment-668118057
api_response = apps_v1.patch_namespaced_deployment_scale(deployment_name, "default", {"spec": {"replicas": 5}})

For troubleshooting purposes, it often makes sense to exec into a Pod and take a look around, possibly grab environment variable to verify correct configuration:

from kubernetes.stream import stream

def pod_exec(name, namespace, command, api_instance):
    exec_command = ["/bin/sh", "-c", command]

    resp = stream(api_instance.connect_get_namespaced_pod_exec,
                  name,
                  namespace,
                  command=exec_command,
                  stderr=True, stdin=False,
                  stdout=True, tty=False,
                  _preload_content=False)

    while resp.is_open():
        resp.update(timeout=1)
        if resp.peek_stdout():
            print(f"STDOUT: \n{resp.read_stdout()}")
        if resp.peek_stderr():
            print(f"STDERR: \n{resp.read_stderr()}")

    resp.close()

    if resp.returncode != 0:
        raise Exception("Script failed")

pod = "example"
api_client = client.ApiClient(configuration)
v1 = client.CoreV1Api(api_client)

pod_exec(pod, "default", "env", v1)

# STDOUT: 
# KUBERNETES_SERVICE_PORT=443
# KUBERNETES_PORT=tcp://10.96.0.1:443
# HOSTNAME=example
# HOME=/root
# ...

The snippet above also allows you to run whole shell scripts if needs be.

Moving onto more cluster administration-oriented tasks - let's say you want to apply a Taint onto a node that has some issue. Well, once again there's no direct API for Node Taints, but we can find a way:

from kubernetes import client

api_client = client.ApiClient(configuration)
v1 = client.CoreV1Api(api_client)

# kubectl taint nodes api-playground-worker some-taint=1:NoSchedule
v1.patch_node("api-playground-worker", {"spec": {"taints": [{"effect": "NoSchedule", "key": "some-taint", "value": "1"}]}})

# kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints --no-headers
# api-playground-control-plane   [map[effect:NoSchedule key:node-role.kubernetes.io/master]]
# api-playground-worker          [map[effect:NoSchedule key:some-taint value:1]]
# api-playground-worker2         <none>
# api-playground-worker3         <none>

You might also want to monitor a cluster resource utilization to possibly automate cluster scaling. Usually, you'd use kubectl top to get the information interactively, with the client library you can do:

# https://github.com/kubernetes-sigs/kind/issues/398
# kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.0/components.yaml
# kubectl patch -n kube-system deployment metrics-server --type=json \
#   -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

from kubernetes import client

api_client = client.ApiClient(configuration)
custom_api = client.CustomObjectsApi(api_client)

response = custom_api.list_cluster_custom_object("metrics.k8s.io", "v1beta1", "nodes")  # also works with "pods" instead of "nodes"  

for node in response["items"]:
    print(f"{node['metadata']['name']: <30} CPU: {node['usage']['cpu']: <10} Memory: {node['usage']['memory']}")

# api-playground-control-plane   CPU: 148318488n Memory: 2363504Ki
# api-playground-worker          CPU: 91635913n  Memory: 1858680Ki
# api-playground-worker2         CPU: 75473747n  Memory: 1880860Ki
# api-playground-worker3         CPU: 105692650n Memory: 1881560Ki

The above example assumes that you have metrics-server installed in your cluster. You can run kubectl top to verify that. Use the comment in the snippet to install it if you're working with KinD.

Last but not least - you might already have a bunch of YAML or JSON files that you want to use to deploy or modify objects in your cluster, or you might want to export and backup what you've created with the client. Here's how you can convert from YAML/JSON files to Kubernetes object and back to files again:

# pip install kopf  # (Python 3.7+)
import kopf

api_client = client.ApiClient(configuration)
v1 = client.CoreV1Api(api_client)

pods = []

# https://stackoverflow.com/questions/59977058/clone-kubernetes-objects-programmatically-using-the-python-api/59977059#59977059
ret = v1.list_namespaced_pod(namespace="default")
for pod in ret.items:
    # Simple conversion to Dict/JSON
    print(api_client.sanitize_for_serialization(pod))

    # Conversion with fields clean-up
    pods.append(kopf.AnnotationsDiffBaseStorage()
                .build(body=kopf.Body(api_client.sanitize_for_serialization(pod))))


# Conversion from Dict back to Client object
class FakeKubeResponse:
    def __init__(self, obj):
        import json
        self.data = json.dumps(obj)


for pod in pods:
    pod_manifest = api_client.deserialize(FakeKubeResponse(pod), "V1Pod")
    ...

First way to convert existing object into Python dictionary (JSON) is to use sanitize_for_serialization which produces raw output with all the generated/default fields. Better option is to use utility methods of kopf library which will remove all the unnecessary fields. From there it's simple enough to convert dictionary into proper YAML or JSON file.

For the reverse - that is if we want to go from dictionary to Client Object Model - we can use deserialize method of API Client. This method however expects its argument to have a data attribute, so we pass it a container class instance with such attribute.

If you already have YAML files which you'd like to use with the Python client, then you can use the utility function kubernetes.utils.create_from_yaml.

To get complete overview of all the features of the library, I recommend you take a look at the examples directory in the repository.

I'd also encourage you to look through the issues in the library repository, as it has a lot of great examples of client usage, such as processing events in parallel or watching ConfigMaps for updates.

Conclusion

The Python client library contains literally hundreds of function, so it's difficult to cover every little feature or use-case there is. Most of them however, follow a common pattern which should make the library's usage pretty natural after couple minutes.

If you're looking for more examples beyond what was shown and referenced above, I recommend exploring other popular tools that make use Python Kubernetes client, such kopf - the library for creating Kubernetes operators. I also find it very useful to take a look at tests of the library itself, as it showcases its intended usage such this client test suite.

End-to-End Monitoring with Grafana Cloud with Minimal Effort

Martin Heinz — Mon, 02 May 2022 21:01:12 +0000

Monitoring is usually at the end of a checklist when building an application, yet it's crucial for making sure that it's running smoothly and that any problems are found and resolved quickly.

Building complete monitoring - including aggregated logging, metrics, tracing, alerting or synthetic probes - requires a lot of effort and time though, not to mention building and managing the infrastructure needed for it. So, in this article we will look at how we can set all of this up quickly and with little effort with no infrastructure needed using Grafana Cloud, all for free.

Infrastructure Provider

Small or medium-sized projects don't warrant spinning up complete monitoring infrastructure, so managed solutions are a good choice. At the same time, no one wants to spend a lot of money just to keep a few microservices alive.

There are however, a couple managed monitoring solutions with free tiers that provide all the things one might need.

First of them being Datadog which provides free infrastructure monitoring, which wouldn't get us very far, especially considering that alerts are not included

Better option is New Relic which has a free plan, which includes probably everything that you might need. One downside is that New Relic uses a set of proprietary tools, which creates a vendor lock and would make it hard to migrate to another platform or to own infrastructure.

Third and in my opinion the best option here is Grafana Cloud which has a quite generous free plan that includes logging, metrics, alerting and synthetic monitoring using a popular set of open source tools such as Prometheus, Alertmanager, Loki and Tempo. This is the best free platform I was able to find and it's what we will use to set up our monitoring.

For a sake of completeness, I also looked Dynatrace, Instana, Splunk, Sumo Logic and AWS Managed Prometheus, those however have no free plans.

As I mentioned, I'm considering only free options. If you need to run a monitoring stack on your infrastructure, I strongly recommend to use "Prometheus and friends", that is Prometheus, Alertmanager, Thanos and Grafana. The easiest deployment option for that is using Prometheus Operator.

Note: This isn't sponsored (heh, I wish), I just decided to take the platform for a spin and I really liked it, so I decided to make write-up, for you and my future self.

Metrics

You can sign up for Grafana Cloud account at https://grafana.com/products/cloud/, assuming you've done so, you should now have your account accessible at https://USERNAME.grafana.net/a/cloud-home-app.

We will start building our monitoring by sending Prometheus metrics to the account. Before we begin though, we need to get access to the Prometheus instance provisioned for us by Grafana Cloud. You can find instances of all the available services in your account on the Grafana Cloud Portal at https://grafana.com/orgs/USERNAME. From there you can navigate to Prometheus configuration by clicking Details button. There you will find all the info needed to send data to your instance, that is username, remote write endpoint and API Key (which you need to generate).

Usually Prometheus scrapes metrics, in Grafana Cloud the Prometheus instance is configured to use push model, where your application has to push metrics using Grafana Cloud Agent. On the above-mentioned Prometheus configuration page you're also presented with sample remote_write configuration which you can add to your agent.

To this out, we will spin up a simple application and agent using docker-compose:

# docker-compose.yml
version: '3.7'

services:
  agent:
    image: grafana/agent:v0.23.0
    container_name: agent
    entrypoint:
      - /bin/agent
      - -config.file=/etc/agent/agent.yaml
      - -metrics.wal-directory=/tmp/wal
      - -config.expand-env
      - -config.enable-read-api
    environment:
      HOSTNAME: agent
      PROMETHEUS_HOST: ${PROMETHEUS_HOST}
      PROMETHEUS_USERNAME: ${PROMETHEUS_USERNAME}
      PROMETHEUS_PASSWORD: ${PROMETHEUS_PASSWORD}
    volumes:
      - ${PWD}/agent/data:/etc/agent/data
      - ${PWD}/agent/config/agent.yaml:/etc/agent/agent.yaml
    ports:
      - "12345:12345"
  api:
    # https://github.com/brancz/prometheus-example-app
    image: quay.io/brancz/prometheus-example-app:v0.3.0
    container_name: api
    ports:
      - 8080:8080
    expose:
      - 9080

The above config provides both the agent and sample Go application with reasonable default settings. It sets the Prometheus host, username and API Key (password) through environment variables which should be provided using .env file.

Along with the above docker-compose.yml, you will also need agent configuration such as:

# agent/config/agent.yaml
server:
  log_level: info
  http_listen_port: 12345

metrics:
  wal_directory: /tmp/wal
  global:
    scrape_interval: 60s
  configs:
    - name: api 
      scrape_configs:
        - job_name: default
          metrics_path: /metrics/
          static_configs:
            - targets: ['api:8080']
      remote_write:
        - basic_auth:
            username: ${PROMETHEUS_USERNAME}
            password: ${PROMETHEUS_PASSWORD}
          url: https://${PROMETHEUS_HOST}/api/prom/push

This config tells the agent to scrape the sample application running at api:8080 with metrics exposed at /metrics. It also tells the agent how to authenticate to Prometheus when pushing metrics.

In a real-world application, you might be inclined to run the /metrics endpoint over HTTPS, that however won't work here, so make sure that the server listens on HTTP, not HTTPS.

Finally, after running docker-compose up you should be able to access the API metrics with curl localhost:8080/metrics and also see logs of agent such as:

agent    | ts=2022-04-15T10:50:06.566528117Z caller=node.go:85 level=info agent=prometheus component=cluster msg="applying config"
agent    | ts=2022-04-15T10:50:06.566605011Z caller=remote.go:180 level=info agent=prometheus component=cluster msg="not watching the KV, none set"
agent    | ts=2022-04-15T10:50:06Z level=info caller=traces/traces.go:135 msg="Traces Logger Initialized" component=traces
agent    | ts=2022-04-15T10:50:06.567958177Z caller=server.go:77 level=info msg="server configuration changed, restarting server"
agent    | ts=2022-04-15T10:50:06.568175733Z caller=gokit.go:72 level=info http=[::]:12345 grpc=[::]:9095 msg="server listening on addresses"
agent    | ts=2022-04-15T10:50:06.57401597Z caller=wal.go:182 level=info agent=prometheus instance=... msg="replaying WAL, this may take a while" dir=/tmp/wal/.../wal
agent    | ts=2022-04-15T10:50:06.574279892Z caller=wal.go:229 level=info agent=prometheus instance=... msg="WAL segment loaded" segment=0 maxSegment=0
agent    | ts=2022-04-15T10:50:06.574886325Z caller=dedupe.go:112 agent=prometheus instance=... component=remote level=info remote_name... url=https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push msg="Starting WAL watcher" queue...
agent    | ts=2022-04-15T10:50:06.57490499Z caller=dedupe.go:112 agent=prometheus instance=... component=remote level=info remote_name... url=https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push msg="Starting scraped metadata watcher"
agent    | ts=2022-04-15T10:50:06.575031636Z caller=dedupe.go:112 agent=prometheus instance=... component=remote level=info remote_name... url=https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push msg="Replaying WAL" queue...
agent    | ts=2022-04-15T10:51:06.155341057Z caller=dedupe.go:112 agent=prometheus instance=... component=remote level=info remote_name... url=https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push msg="Done replaying WAL" duration=59.580374492s

To also view what metrics the agent is itself collecting and sending, you can use curl localhost:12345/metrics.

Dashboards

With the data flowing to your Prometheus instance, it's time to visualize it on a dashboards. Navigate to https://USERNAME.grafana.net/dashboards and click New Dashboard and New Panel in the following screen. Choose grafanacloud-<USERNAME>-prom as a data source and you should see metrics browser field getting populated with your metrics.

Below you can see a sample dashboard showing memory consumption of a Go application. The dashboard was additionally configured to show a threshold of 20MB, which can be set in the bottom of right-side panel.

Synthetics

In addition to monitoring your applications using metrics they expose, you can also leverage synthetic monitoring that allows you to continuously probe the service for status code, response time, DNS resolution, etc.

Grafana Cloud provides synthetic monitoring which can be accessed at https://USERNAME.grafana.net/a/grafana-synthetic-monitoring-app/home. From there you can navigate to Checks tab and click Add new check. There you can choose from HTTP, PING, DNS, TCP or Traceroute options which are explained in docs. Filling out the remaining fields should be pretty self-explanatory, when choosing probe location though, be aware that the more probes you run the more logs will be produced which count towards your consumption/usage limit.

Nice feature of Grafana Cloud Synthetics is that you can export them as Terraform configuration, so you can build the checks manually via UI and get the configuration as a code. To do so navigate to https://USERNAME.grafana.net/plugins/grafana-synthetic-monitoring-app and click Generate config button.

After you are done creating your checks you can view dashboards that are automatically created for each type of check. For example HTTP check dashboard would look like this:

If you're monitoring website that has web analytics configured, then you will want to exclude IPs of Grafana probes from analytics collection. There's unfortunately no list of static IPs, however you can use the following DNS names to find the IPs.

Alerts and Notifications

Both metrics and synthetic probes provide us with plenty of data which we can use to create alerts. To create new alerts you can navigate to https://USERNAME.grafana.net/alerting/list and click New alert rule. If you followed the above example with Prometheus, then you should choose grafanacloud-<USERNAME>-prom as data source. You should already see 2 query fields prepared, you can put your metric in the field A and the expression to evaluate the rule in field B. The complete configuration should look like so:

When you scrolled through the available metrics you might have noticed that it now also includes fields such as probe_all_success_sum (generally, probe_*), these are metrics generated by synthetic monitors shown in previous section and you can create alerts using those too. Some useful examples would be:

SSL Expiration:

probe_ssl_earliest_cert_expiry{instance="https://some-url.com", job="Ping Website", probe="Frankfurt"} - time()

# Use Condition: WHEN: last() OF A IS BELOW (86400 * N Days)

API Availability:

sum(
  (
    increase(probe_all_success_sum{instance="some-url.com", job="Ping API"}[5m])
    OR
    increase(probe_success_sum{instance="some-url.com", job="Ping API"}[5m])
  )
)
/
sum(
  (
    increase(probe_all_success_count{instance="some-url.com", job="Ping API"}[5m])
    OR
    increase(probe_success_count{instance="some-url.com", job="Ping API"}[5m])
  )
)

# Use condition WHEN avg() OF A IS BELOW 0.9X

Ping Success Rate:

avg_over_time(probe_all_success_count{instance="some-url.com"}[5m])
/
avg_over_time(probe_all_success_sum{instance="some-url.com"}[5m])

# Use condition WHEN avg() OF A IS BELOW 0.9X

Agent Watchdog:

up{instance="backend:9080"}

# Use condition WHEN last() OF A IS BELOW 1

You can also use the existing synthetic monitoring dashboards for inspiration - when you hover over any panel in the dashboard, you will see the PromQL query used to create it. You can also go into dashboard settings, make it editable and then copy the queries from each panel.

With alerts ready, we need to configure a Contact points to which they will send notifications. Those can be viewed at https://USERNAME.grafana.net/alerting/notifications. By default, there's an email contact point created for you, you should however populate its email address, otherwise you won't receive the notifications. You can add more contact points by clicking New contact point.

Finally, to route the alert to the correct contact point, we need to configure Notification policies at https://USERNAME.grafana.net/alerting/routes, otherwise it would all go the default email contact. Click New Policy and set the contact point of your choosing, optionally also set matching labels if you want to assign only a subset of alerts to this contact. In my case, I set the label to channel=slack both on alerts and the policy.

And here are the resulting alerts send to email and Slack respectively. If you're not a fan of the default template of the alert messages, you can add your own in Contact points tab by clicking New template.

Logs

Grafana Cloud also allows you to collect logs using Loki. Instance of which is automatically provisioned for you. To start sending logs from your applications to the Grafana Cloud, you can head to the Grafana Cloud Portal and retrieve credentials for Loki same as with Prometheus earlier.

You can either use the agent to send the logs or if you're running the apps with Docker, then you can use Loki logging plugin, which we will do here.

First you will need to install the plugin:

docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
docker plugin ls
ID             NAME          DESCRIPTION           ENABLED
605d80959327   loki:latest   Loki Logging Driver   true

After which you need to update the docker-compose.yml like so:

version: '3.7'

+x-logging:
+  &default-logging
+  driver: loki
+  options:
+    loki-url: "https://${LOKI_USERNAME}:${LOKI_PASSWORD}@${LOKI_HOST}/loki/api/v1/push"

services:
  agent:
    image: grafana/agent:v0.23.0
    container_name: agent
+    logging: *default-logging
    entrypoint:
      - ...
    environment:
      ...
+      LOKI_HOST: ${LOKI_HOST}
+      LOKI_USERNAME: ${LOKI_USERNAME}
+      LOKI_PASSWORD: ${LOKI_PASSWORD}
    volumes:
      - ${PWD}/agent/data:/etc/agent/data
      - ${PWD}/agent/config/agent.yaml:/etc/agent/agent.yaml
    ports:
      - "12345:12345"
  api:
    image: quay.io/brancz/prometheus-example-app:v0.3.0
+    logging: *default-logging
    container_name: api
    ports:
      - 8080:8080
    expose:
      - 9080

The above sets the logging driver provided by the Docker Loki plugin for each container. If you want to set the driver for individual containers started with docker run, then checkout docs here.

After you start your containers with updated config, you can verify whether agent found logs by checking /tmp/positions.yaml file inside the agent container.

With logs flowing to Grafana Cloud you can view them in https://USERNAME.grafana.net/explore by choosing grafanacloud-USERNAME-logs as a data source and querying your project and containers:

Traces

Final piece of puzzle in the monitoring setup would be traces using Tempo. Again, similarly to Prometheus and Loki config, you can grab credentials and sample agent configuration for Tempo from Grafana Cloud Portal.

The additional config you will need for the agent should look like:

# All config options at https://grafana.com/docs/agent/latest/configuration/traces-config/
traces:
  configs:
    - name: default
      remote_write:
        - endpoint: ${TEMPO_HOST}:443
          basic_auth:
            username: ${TEMPO_USERNAME}
            password: ${TEMPO_PASSWORD}
      receivers:
        otlp:
          protocols:
            http:

Additionally docker-compose.yml and .env should include credentials variables for TEMPO_HOST, TEMPO_USERNAME and TEMPO_PASSWORD. After restarting your containers, you should see in agent logs that the tracing component got initialized:

agent | ... msg="Traces Logger Initialized" component=traces
agent | ... msg="shutting down receiver" component=traces traces_config=default
agent | ... msg="shutting down processors" component=traces traces_config=default
agent | ... msg="shutting down exporters" component=traces traces_config=default
agent | ... msg="Exporter was built." component=traces traces_config=default kind=exporter name=otlp/0
agent | ... msg="Exporter is starting..." component=traces traces_config=default kind=exporter name=otlp/0
agent | ... msg="Exporter started." component=traces traces_config=default kind=exporter name=otlp/0
agent | ... msg="Pipeline was built." component=traces traces_config=default name=pipeline name=traces
agent | ... msg="Pipeline is starting..." component=traces traces_config=default name=pipeline name=traces
agent | ... msg="Pipeline is started." component=traces traces_config=default name=pipeline name=traces
agent | ... msg="Receiver was built." component=traces traces_config=default kind=receiver name=otlp datatype=traces
agent | ... msg="Receiver is starting..." component=traces traces_config=default kind=receiver name=otlp
agent | ... msg="Starting HTTP server on endpoint 0.0.0.0:4318" component=traces traces_config=default kind=receiver name=otlp
agent | ... msg="Setting up a second HTTP listener on legacy endpoint 0.0.0.0:55681" component=traces traces_config=default kind=receiver name=otlp
agent | ... msg="Starting HTTP server on endpoint 0.0.0.0:55681" component=traces traces_config=default kind=receiver name=otlp
agent | ... msg="Receiver started." component=traces traces_config=default kind=receiver name=otlp

In the above output you can see that the collector is listening for traces at 0.0.0.0:4318. You should therefore configure your application to send the telemetry data to this endpoint. See OpenTelemetry reference to find variables relevant for your SDK.

To verify that traces are being collected and send to Tempo, you can run curl -s localhost:12345/metrics | grep traces, which will show you following metrics:

# TYPE traces_exporter_enqueue_failed_log_records counter
traces_exporter_enqueue_failed_log_records{exporter="otlp/0",traces_config="default"} 0
# HELP traces_exporter_enqueue_failed_metric_points Number of metric points failed to be added to the sending queue.
# TYPE traces_exporter_enqueue_failed_metric_points counter
traces_exporter_enqueue_failed_metric_points{exporter="otlp/0",traces_config="default"} 0
# HELP traces_exporter_enqueue_failed_spans Number of spans failed to be added to the sending queue.
# TYPE traces_exporter_enqueue_failed_spans counter
traces_exporter_enqueue_failed_spans{exporter="otlp/0",traces_config="default"} 0
# HELP traces_exporter_queue_size Current size of the retry queue (in batches)
# TYPE traces_exporter_queue_size gauge
traces_exporter_queue_size{exporter="otlp/0",traces_config="default"} 0

Closing Thoughts

Initially I was somewhat annoyed by the 15-day Pro Trial, as I wanted to test the limits of the Free plan. However, after the trial expired I realised that I didn't even exceed the free plan limit and didn't touch the Pro/paid features, so the Free plan seems to be quite generous, especially if you're just trying to monitor a couple of microservices with low-ish traffic.

Even with the free plan Grafana Cloud really provides all the tools you need to set up complete monitoring at actual zero cost for reasonably large deployments. I also really like that the usage is calculated only based on amount of metric series and log lines, making it very easy to track. This is also helped by comprehensive Billing/Usage dashboards.

Goodbye, Google Analytics - Why and How You Should Leave The Platform

Martin Heinz — Tue, 19 Apr 2022 18:06:57 +0000

With the recent events relating to Google Analytics platform, it's becoming very clear that the time has come for many of us to migrate from Google Analytics to different platforms.

In this article we will go over both the "Why?", so that you can make an informed decision whether you need to migrate of not, as well as the "How?" of migrating from Google Analytics - that is quickly and easily taking your data and moving to different analytics platform without too much hassle.

Why?

Why has now - all of a sudden - came time to ditch Google Analytics?

There are 2 main reasons why you might want to do so. First being that there have been court rulings in multiple EU countries stating that it's illegal to use Google Analytics in EU. This is because of data of EU citizens is being transferred to US, which violates GDPR rules. These ruling apply so far to France and Austria, but more are expected in 2022. So, unless Google decides to run GA infrastructure in EU pretty soon, then you might be left in legal hot water pretty soon.

Recently, US and EU leadership reached agreement on trans-Atlantic data privacy, which might alleviate this issue, but chances are that data privacy advocates won't be satisfied and will challenge this in courts.

The other reason to migrate off the platform is that Google is deprecating Universal Analytics beginning 1 July 2023. You could migrate your GA properties to GA4 as advised by Google, the problem here's though that your old data won't move and you won't be able to compare or look back at analytics from let's say 2 years ago, which sucks.

The solution that "kills two birds with one stone" is to choose a privacy-friendly platform where you won't need to worry about GDPR rules and extract and move your data from GA to said platform, taking full ownership of your data and not having to worry about Google deprecating things whenever they please (which is very often).

How?

If you decided that you want to or need to migrate, then here I present you a simple tool and process for exporting your analytics data and getting it into a format suitable for other platforms.

GA Extractor can help you extract any analytics data from Google Analytics without having to touch the API. Apart from performing basic data exports, it can also transform it into other formats besides default JSON - that being - a more readable CSV, which can be used in data analysis or SQL suitable for direct migration to other platforms.

Let's look at some examples. To install the tool just run:

pip install ga-extractor
ga-extractor --help
# Usage: ga-extractor [OPTIONS] COMMAND [ARGS]...
# ...

From there you can set it up like so:

ga-extractor setup \
  --sa-key-path="analytics-api-24102021-4edf0b7270c0.json" \
  --table-id="123456789" \
  --metrics="ga:sessions" \
  --dimensions="ga:browser" \
  --start-date="2022-03-15" \
  --end-date="2022-03-19"

In the command above, you can see that you need to provide Service Account key file (--sa-key-path) and Table ID (--table-id, also referred to as View ID). You get these values using the guide in repository README or on PyPI page.

Running this command will generate a config file in an application config directory in ~/.config/ga-extractor/.... You can verify that the tool can authenticate with Google API by running ga-extractor auth.

If you're unsure as to which metrics and dimensions to select, then you can use --preset flag with FULL or BASIC option to set the metrics and dimensions for you.

With tool configured, you can now run extraction:

ga-extractor extract --report="my-awesome-report.json"
# Report written to /home/user/.config/ga-extractor/my-awesome-report.json

cat /home/user/.config/ga-extractor/my-awesome-report.json | jq .
# ...

Now, in case the data format produced by Google API isn't suitable for your needs, then you can use migrate command to extract and transform it into CSV or SQL:

ga-extractor migrate --format=CSV
# Report written to /home/user/.config/ga-extractor/02c2db1a-1ff0-47af-bad3-9c8bc51c1d13_extract.csv

head /home/user/.config/ga-extractor/02c2db1a-1ff0-47af-bad3-9c8bc51c1d13_extract.csv
# path,browser,os,device,screen,language,country,referral_path,count,date
# /,Chrome,Android,mobile,1370x1370,zh-cn,China,(direct),1,2022-03-18
# /,Chrome,Android,mobile,340x620,en-gb,United Kingdom,t.co/,1,2022-03-18

ga-extractor migrate --format=UMAMI
# Report written to /home/user/.config/ga-extractor/cee9e1d0-3b87-4052-a295-1b7224c5ba78_extract.sql

The CSV is much more readable than the default output, while the SQL can be used to migration to other platforms. The SQL here is tailored for Umami Analytics, but with some tweaks it could be applied to other platforms that track sessions and page views along with all the common metrics.

Be aware that the migrate command overrides previously configured metrics and dimensions, so that it can produce meaningful results.

If you don't have a lot of data or you feel like the above is too much hassle for you, then you can also export some tables and views from GA web console. There are a lot of advanced/complex things you can export, but 2 most basic things you surely want take before leaving GA is page views and user sessions information.

To export page view count or really any aggregated metric, you can navigate to Audience and Overview, choose desired metric from dropdown and click export:

To export individual sessions you can navigate to Audience and User Explorer, and click export:

There are many more things you can grab from the console, and I recommend checking out this article which provides good overview.

Alternatives

Now that you have the data exported, or you maybe decided to leave GA behind without even taking the data, it's time to choose the new platform. There are many good privacy-friendly, open source, cheap/free GA alternatives. I won't go over every single one of these as there are great articles written about this, like this one with a list of open-source options.

If you exported data using the ga-exporter tool presented above you have the option to transform it into format that can be readily inserted into Umami which makes it easy migration target. If you want to test it out, then you can spin up the service with Docker as described in docs, add your website as shown here and then just insert previously exported data with:

cat  /home/user/.config/ga-extractor/cee9e1d0-3b87-4052-a295-1b7224c5ba78_extract.sql | psql -Upostgres -a db

You can then view the dashboard, which should show you your data like so:

Umami is my migration target, you might however have different preferences and needs. One thing to keep in mind though, when choosing your target platform, is full data "ownership", which is important to avoid vendor lock, which might otherwise make it hard to migrate in the future, if the need arises again. So, I'd recommend to look for a platform that makes it easy to export full analytics data in practical format(s) to make it easy to analyze your data or migrate elsewhere.

Closing Thoughts

Even if you decide that you don't want to or don't need to leave Google Analytics at this time, it's good idea to explore some of the alternatives. Chances are that you don't need any of the advanced GA features, and you might just realize that the alternatives are actually more suitable for your needs.

There's also no need to be scared of self-hosting the analytics engine yourself. Many of the open-source solutions can be spun up in matter of minutes and require very little resources to run. Even data migration can be quite simple as you've seen earlier in this article. If that's the route you want to go, but the extractor tool presented here doesn't support the target platform you'd like to migrate to, or you have some feedback to share, then feel free to create and an issue in GitHub repository and I will definitely try to help out.

Python f-strings Are More Powerful Than You Might Think

Martin Heinz — Mon, 04 Apr 2022 19:15:47 +0000

Formatted string literals - also called f-strings - have been around since Python 3.6, so we all know what they are and how to use them. There are however some facts and handy features of f-string that you might not know about. So, let's take a tour of some awesome f-string features that you'll want to use in your everyday coding.

Date and Time Formatting

Applying number formatting with f-strings is pretty common, but did you know that you can also format dates and timestamp strings?

import datetime
today = datetime.datetime.today()
print(f"{today:%Y-%m-%d}")
# 2022-03-11
print(f"{today:%Y}")
# 2022

f-strings can format date and time as if you used datetime.strftime method. This is extra nice, when you realize that there are more formats than just the few mentioned in the docs. Python's strftime supports also all the formats supported by the underlying C implementation, which might vary by platform and that's why it's not mentioned in docs. With that said you can take advantage of these formats anyway and use for example %F, which is an equivalent of %Y-%m-%d or %T which is an equivalent of %H:%M:%S, also worth mentioning are %x and %X which are locales preferred date and time formats respectively. Usage of these formats is obviously not limited to f-strings. Refer to the Linux manpages for full list of formats.

Variable Names and Debugging

One of the more recent additions to f-string features (starting with Python 3.8) is ability to print variable names along with the value:

x = 10
y = 25
print(f"x = {x}, y = {y}")
# x = 10, y = 25
print(f"{x = }, {y = }")  # Better! (3.8+)
# x = 10, y = 25

print(f"{x = :.3f}")
# x = 10.000

This feature is called "debugging" and can be applied in combination with other modifiers. It also preserves whitespaces, so f"{x = }" and f"{x=}" will produce different strings.

`repr` and `str`

When printing class instances, __str__ method of the class is used by default for string representation. If we however want to force usage of __repr__, we can use the !r conversion flag:


class User:
    def __init__(self, first_name, last_name):
        self.first_name = first_name
        self.last_name = last_name

    def __str__(self):
        return f"{self.first_name} {self.last_name}"

    def __repr__(self):
        return f"User's name is: {self.first_name} {self.last_name}"

user = User("John", "Doe")
print(f"{user}")
# John Doe
print(f"{user!r}")
# User's name is: John Doe

We could also just call repr(some_var) inside the f-string, but using the conversion flag is a nice native and concise solution.

Superior Performance

Powerful features and syntax sugar oftentimes comes with performance penalty, that's however not the case when it comes to f-strings:

# python -m timeit -s 'x, y = "Hello", "World"' 'f"{x} {y}"'
from string import Template

x, y = "Hello", "World"

print(f"{x} {y}")  # 39.6 nsec per loop - Fast!
print(x + " " + y)  # 43.5 nsec per loop
print(" ".join((x, y)))  # 58.1 nsec per loop
print("%s %s" % (x, y))  # 103 nsec per loop
print("{} {}".format(x, y))  # 141 nsec per loop
print(Template("$x $y").substitute(x=x, y=y))  # 1.24 usec per loop - Slow!

The above samples were tested with timeit module like so: python -m timeit -s 'x, y = "Hello", "World"' 'f"{x} {y}"' and as you can see f-strings are actually the fastest of all formatting options Python provides. So, even if you prefer using some of the older formatting options, you might consider switching to f-strings just for the performance boost.

Full Power of Formatting Spec

F-strings support Python's Format Specification Mini-Language, so you can embed a lot of formatting operations into their modifiers:

text = "hello world"

# Center text:
print(f"{text:^15}")
# '  hello world  '

number = 1234567890
# Set separator
print(f"{number:,}")
# 1,234,567,890

number = 123
# Add leading zeros
print(f"{number:08}")
# 00000123

Python's formatting mini-language includes much more than just the options to format numbers and dates. It allows us to align or center text, add leading zeros/spaces, set thousands separator and more. All this is obviously available not just for f-strings, but for all the other formatting options too.

Nested F-Strings

If basic f-strings aren't good enough for your formatting needs you can even nest them into each other:

number = 254.3463
print(f"{f'${number:.3f}':>10s}")
# '  $254.346'

You can embed f-strings inside f-strings for tricky formatting problems like adding a dollar sign to a right aligned float, as shown above.

Nested f-strings can also be used in case you need to use variables in the format specifier part. This can also make the f-string more readable:

import decimal
width = 8
precision = 3
value = decimal.Decimal("42.12345")
print(f"output: {value:{width}.{precision}}")
# 'output:     42.1'

Conditionals Formatting

Building on top of the above example with nested f-strings, we can go a bit farther and use ternary conditional operators inside the inner f-string:

import decimal
value = decimal.Decimal("42.12345")
print(f'Result: {value:{"4.3" if value < 100 else "8.3"}}')
# Result: 42.1
value = decimal.Decimal("142.12345")
print(f'Result: {value:{"4.2" if value < 100 else "8.3"}}')
# Result:      142

This can become very unreadable very quickly, so you might want to break it into multiple lines instead.

Lambda Expressions

If you want to push limits of f-strings and also make whoever reads your code angry, then - with a little bit of effort - you can also use lambdas:

print(f"{(lambda x: x**2)(3)}")
# 9

Parenthesis around the lambda expression are in this case mandatory, because of the :, which would be otherwise interpreted by f-string.

Closing Thoughts

As we've seen here, f-strings really are quite powerful and have many more features than most people think. Most of these "unknown" features are however mentioned in Python docs, so I do recommend reading through docs pages of not just f-strings, but any other module/feature of Python you might be using. Diving into the docs will oftentimes help you uncover some very useful features that you won't find even when digging through StackOverflow.

Ultimate CI Pipeline for All of Your Python Projects

Martin Heinz — Tue, 15 Mar 2022 19:15:12 +0000

Every project can benefit from a robust continuous integration pipeline that builds your application, runs tests, lints code, verifies code quality, runs vulnerability analysis and more. However, building such pipeline takes a significant amount of time, which doesn't really provide any benefit on its own. So, if you want a fully featured, customizable CI pipeline for your Python project based on GitHub Actions with all the tools and integrations you could think of, ready in about 5 minutes, then this article has you covered!

Quick Start

If you're not very patient or just want to get rolling right away, then here's minimal setup needed to get the pipeline up and running:

# .github/workflows/python-pipeline.yml
name: CI Pipeline

on: [push, workflow_dispatch]

jobs:
  python-ci-pipeline:
    uses: MartinHeinz/workflows/.github/workflows/python-container-ci.yml@v1.0.0
    with:
      PYTHON_VERSION: '3.10'
      DEPENDENCY_MANAGER: 'pip'
      ENABLE_SONAR: ${{ false }}
      ENABLE_CODE_CLIMATE: ${{ false }}
      ENABLE_SLACK: ${{ false }}
      ENFORCE_PYLINT: ${{ false }}
      ENFORCE_BLACK: ${{ false }}
      ENFORCE_FLAKE8: ${{ false }}
      ENFORCE_BANDIT: ${{ false }}
      ENFORCE_DIVE: ${{ false }}

The above YAML configures a GitHub Actions workflow that references a reusable workflow from my repository here. This way you don't need to copy (and later maintain) the large YAML with all the actions. All you need to do is include this YAML in .github/workflows/ directory of you repository, and configure parameters outlined in the with: stanza to your liking.

All these parameters (config options) have sane defaults and none of them are required, so you can omit the whole with stanza if you trust my judgement. If you don't, then you can tweak them as shown above, along with the remaining options shown in the workflow definition here. For explanation as to how you can find values and configure the secrets required for example for Sonar or CodeClimate integrations - see README in the repository, or read through the following sections where it's explained in detail.

The workflow obviously has to make some assumptions about contents of your repository, so it's expected that there's a source code directory of your application and Dockerfile, with everything else being optional. For a sample of how the repository layout might look like see testing repository here.

As you probably noticed, the above snippet references a specific version of the workflow using @v1.0.0. This is to avoid pulling in any potential changes you might not want. With that said, do checkout the repository from time-to-time as there might be some additional changes and improvement, and therefore new releases.

The Basics

You might have the pipeline configured by now, but let's actually look at what it does on the inside and how you can further customize it. We start with the obvious basics - checking out the repository, installing code and running tests:

jobs:
    # ... Trimmed for clarity
    - uses: actions/checkout@v1
    - uses: actions/setup-python@v1
      id: setup-python
      with:
        python-version: ${{ inputs.PYTHON_VERSION }}
    - name: Get cache metadata
      id: cache-meta
      run: |
        # ...
        # Find cache key and cache path based on dependency manager and its lock file
    - name: Load cached venv
      id: cache
      uses: actions/cache@v2
      with:
        path: ${{ steps.cache-meta.outputs.cache-path }}
        key: ${{ steps.cache-meta.outputs.cache-key }}
    - name: Install Dependencies
      run: |
        # ...
        # Create virtual environment, install dependencies using the configured dependency manager
    - name: Run Tests
      run: |
        source venv/bin/activate
        pytest

The above snippet is trimmed for sake of clarity, but all the steps should be fairly obvious if you're familiar with GitHub Actions. If not, don't worry, you don't actually need to understand this, what's important to know is that the pipeline works with all major Python dependency managers - that is pip poetry and pipenv, all you need to do is set DEPENDENCY_MANAGER and pipeline will take care of the rest. This obviously assumes that you have requirements.txt, poetry.lock or Pipfile.lock in your repository.

The steps above also create Python's virtual environment which is used throughout the pipeline to create isolated build/test environment. This also allows us to cache all dependencies to shave off some time from pipeline runtime. As a bonus, the dependencies are cached across branches as long as the lock file doesn't change.

As for testing step - pytest is used to run your test suite. Pytest will automatically pick up whatever configuration you might have in the repository (if any), more specifically in order of precedence: pytest.ini, pyproject.toml, tox.ini or setup.cfg.

Code Quality

Beyond the basics, we will also want to enforce some code quality measures. There's a lot of code quality tools that you can use to ensure your Python code is clean and maintainable and this pipeline includes, well, all of them:

    - name: Verify code style (Black)
      uses: psf/black@stable
      with:
        options: "--verbose ${{ inputs.ENFORCE_BLACK && '--check' || '' }}"

    - name: Enforce code style (Flake8)
      run: |
        source venv/bin/activate
        flake8 ${{ inputs.ENFORCE_FLAKE8 && '' || '--exit-zero' }}

    - name: Lint code
      run: |
        source venv/bin/activate
        pylint **/*.py  # ... Some more conditional arguments

    - name: Send report to CodeClimate
      uses: paambaati/codeclimate-action@v3.0.0
      if: ${{ inputs.ENABLE_CODE_CLIMATE }}
      env:
        CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}
      with:
        coverageLocations: |
          ${{github.workspace}}/coverage.xml:coverage.py

    - name: SonarCloud scanner
      uses: sonarsource/sonarcloud-github-action@master
      if: ${{ inputs.ENABLE_SONAR }}
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

We start by running Black - the Python code formatter. It's a best practice to use Black as a pre-commit hook or to run it everytime you save a file locally, so this step should serve only as a verification that nothing has slipped through the cracks.

Next, we run Flake8 and Pylint, which apply further style and linting rules beyond what Black does. Both of these are configurable through their respective config files, which will be recognized automatically. In case of Flake8 the options are: setup.cfg, tox.ini, or .flake8 and for Pylint: pylintrc, .pylintrc, pyproject.toml.

All of the above tools can be set to enforcing mode which will make the pipeline fail if issues are found. The config options for these are: ENFORCE_PYLINT, ENFORCE_BLACK and ENFORCE_FLAKE8.

Aside from Python specific tooling, the pipeline also includes 2 popular external tools - which are SonarCloud and CodeClimate. Both are optional, but I do recommend using them, considering that they're available for any public repository. If turned on (using ENABLE_SONAR and/or ENABLE_CODE_CLIMATE), Sonar scanner will run code analysis of you code and send it the SonarCloud and CodeClimate will take code coverage report generated during pytest invocation and will generate coverage report for you.

To configure each of these tools, include their respective config fields in your pipeline config:

jobs:
  python-ci-pipeline:
    uses: MartinHeinz/workflows/.github/workflows/python-container-ci.yml@v1.0.0
    with:
      # ...
      ENABLE_SONAR: ${{ true }}
      ENABLE_CODE_CLIMATE: ${{ true }}
    secrets:
      SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
      CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}

And follow the steps outlined in repository README to generate and set the values for these secrets.

Package

When we're confident that our code is up to standard, it's time package it - in this case in form of container image:

    - name: Login to GitHub Container Registry
      uses: docker/login-action@v1
      with:
        # ... Set registry, username and password

    - name: Generate tags and image meta
      id: meta
      uses: docker/metadata-action@v3
      with:
        images: |
           ${{ inputs.CONTAINER_REGISTRY }}/${{ steps.get-repo.outputs.repo }}
        tags: |
          type=ref,event=tag
          type=sha

    - name: Build image
      uses: docker/build-push-action@v2
      with:
        context: .
        load: true  # Do not push
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        cache-from: type=registry,ref=${{ inputs.CONTAINER_REGISTRY }}/${{ steps.get-repo.outputs.repo }}:latest
        cache-to: type=registry,ref=${{ inputs.CONTAINER_REGISTRY }}/${{ steps.get-repo.outputs.repo }}:latest,mode=max

    - name: Analyze image efficiency
      uses: MartinHeinz/dive-action@v0.1.3
      with:
        image: ${{ inputs.CONTAINER_REGISTRY }}/${{ steps.get-repo.outputs.repo }}:${{ steps.meta.outputs.version }}
        config: ${{ inputs.DIVE_CONFIG }}
        exit-zero: ${{ !inputs.ENFORCE_DIVE }}

    - name: Push container image
      uses: docker/build-push-action@v2
      with:
        push: true
        # ... Rest same as during build

This pipeline defaults to using GitHub Container Registry which is part of your repository. If that's what you want to use, you don't need to configure anything, apart from providing Dockerfile in the repository.

If you prefer to use Docker Hub or any other registry, you can provide CONTAINER_REGISTRY and CONTAINER_REPOSITORY along with credential in CONTAINER_REGISTRY_USERNAME and CONTAINER_REGISTRY_PASSWORD (in secrets stanza of pipeline config) and pipeline will take care of the rest.

Apart from the basic login, build and push sequence, this pipeline also generates additional metadata information which is attached to the image. This includes tagging the image with commit SHA, as well as git tag if there's one.

To make the pipeline extra speedy, cache is also used during docker build to avoid creating image layers that don't need to be rebuilt. Finally, for additional efficiency also Dive tool is ran against the image to evaluate efficiency of the image itself. It also gives you an option to provide config file in .dive-ci and set thresholds for Dive's metrics. As with all the other tools in this pipeline, Dive can also be set to enforcing/non-enforcing mode using ENFORCE_DIVE.

Security

Let's not forget that CI pipeline should make sure that our code doesn't contain any vulnerabilities. For that purpose this workflow includes additional couple of tools:

    - name: Code security check
      run: |
        # ...
        bandit -r . --exclude ./venv  # ... Some more conditional arguments
    - name: Trivy vulnerability scan
      uses: aquasecurity/trivy-action@master
      with:
        image-ref: '${{ inputs.CONTAINER_REGISTRY }}/${{ steps.get-repo.outputs.repo }}:${{ steps.meta.outputs.version }}'
        format: 'sarif'
        output: 'trivy-results.sarif'
    - name: Upload Trivy scan results to GitHub Security tab
      uses: github/codeql-action/upload-sarif@v1
      with:
        sarif_file: 'trivy-results.sarif'
    - name: Sign the published Docker image
      env:
        COSIGN_EXPERIMENTAL: "true"
      run: cosign sign ${{ inputs.CONTAINER_REGISTRY }}/${{ steps.get-repo.outputs.repo }}:${{ steps.meta.outputs.version }}

First of these is a Python tool called Bandit, which looks for common security issues in Python code. This tool has default set of rules, but can be tweaked with config file specified in BANDIT_CONFIG option of the workflow. As other tools mentioned earlier, Bandit also by default runs in non-enforcing mode, but can switch to enforcing with ENFORCE_BANDIT option.

Another tool included in this pipeline that checks for vulnerabilities is Trivy by Aqua Security, which scans the container image and generates a list of vulnerabilities found in the image itself, which extends past the issue limited to your Python code. This report is then uploaded to GitHub Code Scanning which will then show up in your repository's Security tab:

It's great that the above tools assure security of the application we've built, but we should also provide proof of authenticity of the final container image to avoid supply chain attacks. To do so, we use cosign tool to sign the image with GitHub's OIDC token which ties the image to the identity of the user that pushed the code to the repository. This tool doesn't require any key to generate the signature, so this will work out-of-the box. This signature is then pushed to the container registry along with your image - for example in Docker Hub:

And the above signature can be then verified using:

docker pull ghcr.io/some-user/some-repo:sha-1dfb324
COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/some-user/some-repo:sha-1dfb324

# If you don't want to install cosign...
docker run -e COSIGN_EXPERIMENTAL=1 --rm \
  gcr.io/projectsigstore/cosign:v1.6.0 verify \
  ghcr.io/some-user/some-repo:sha-1dfb324

...
Verification for ghcr.io/some-user/some-repo:sha-1dfb324 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - Any certificates were verified against the Fulcio roots.

For more information about cosign and container image signing see article on GitHub's blog.

Notify

Final little feature of this pipeline is a Slack notification, which runs both for successful and failing builds - assuming you turn it on with ENABLE_SLACK. All you need to provide is Slack channel webhook using SLACK_WEBHOOK repository secret.

To generate the said webhook, follow the notes in README.

Closing Thoughts

That should be everything you need to get your end-to-end fully configured pipeline up-and-running.

If the customization options or features doesn't exactly fit your needs, feel free to fork the repository or submit an issue with a feature request.

Apart from additional features, more pipeline options for Python (or other languages) might appear in this repository in the future, as the pipeline presented here won't fit every type of application. So, in case you're interested in those or any new development, make sure to check out the repository from time-to-time to find out about any new releases.

Optimizing Memory Usage of Python Applications

Martin Heinz — Mon, 28 Feb 2022 19:51:52 +0000

When it comes to performance optimization, people usually focus only on speed and CPU usage. Rarely is anyone concerned with memory consumption, well, until they run out of RAM. There are many reasons to try to limit memory usage, not just avoiding having your application crash because of out-of-memory errors.

In this article we will explore techniques for finding which parts of your Python applications are consuming too much memory, analyze the reasons for it and finally reduce the memory consumption and footprint using simple tricks and memory efficient data structures.

Why Bother, Anyway?

But first, why should you bother saving RAM anyway? Are there really any reason to save memory other than avoiding the aforementioned out-of-memory errors/crashes?

One simple reason is money. Resources - both CPU and RAM - cost money, why waste memory by running inefficient applications, if there are ways to reduce the memory footprint?

Another reason is the notion that "data has mass", if there's a lot of it, then it will move around slowly. If data has to be stored on disk rather than in RAM or fast caches, then it will take a while to load and get processed, impacting overall performance. Therefore, optimizing for memory usage might have a nice side effect of speeding-up the application runtime.

Lastly, in some cases performance can be improved by adding more memory (if application performance is memory-bound), but you can't do that if you don't have any memory left on the machine.

Find Bottlenecks

It's clear that there are good reasons to reduce memory usage of our Python applications, before we do that though, we first need to find the bottlenecks or parts of code that are hogging all the memory.

First tool we will introduce is memory_profiler. This tool measures memory usage of specific function on line-by-line basis:

# https://github.com/pythonprofilers/memory_profiler
pip install memory_profiler psutil
# psutil is needed for better memory_profiler performance

python -m memory_profiler some-code.py
Filename: some-code.py

Line #    Mem usage    Increment  Occurrences   Line Contents
============================================================
    15   39.113 MiB   39.113 MiB            1   @profile
    16                                          def memory_intensive():
    17   46.539 MiB    7.426 MiB            1       small_list = [None] * 1000000
    18  122.852 MiB   76.312 MiB            1       big_list = [None] * 10000000
    19   46.766 MiB  -76.086 MiB            1       del big_list
    20   46.766 MiB    0.000 MiB            1       return small_list

To start using it, we install it with pip along with psutil package which significantly improves profiler's performance. In addition to that, we also need to mark the function we want to benchmark with @profile decorator. Finally, we run the profiler against our code using python -m memory_profiler. This shows memory usage/allocation on line-by-line basis for the decorated function - in this case memory_intensive - which intentionally creates and deletes large lists.

Now that we know how to narrow down our focus and find specific lines that increase memory consumption, we might want to dig a little deeper and see how much each variable is using. You might have seen sys.getsizeof used to measure this before. This function however will give you questionable information for some types of data structures. For integers or bytearrays you will get the real size in bytes, for containers such as list though, you will only get size of the container itself and not its contents:

import sys
print(sys.getsizeof(1))
# 28
print(sys.getsizeof(2**30))
# 32
print(sys.getsizeof(2**60))
# 36

print(sys.getsizeof("a"))
# 50
print(sys.getsizeof("aa"))
# 51
print(sys.getsizeof("aaa"))
# 52

print(sys.getsizeof([]))
# 56
print(sys.getsizeof([1]))
# 64
print(sys.getsizeof([1, 2, 3, 4, 5]))
# 96, yet empty list is 56 and each value inside is 28.

We can see that with plain integers, everytime we cross a threshold, 4 bytes are added to the size. Similarly, with plain strings, everytime we add another character one extra byte is added. With lists however, this doesn't hold up - sys.getsizeof doesn't "walk" the data structure and only returns size of the parent object, in this case list.

Better approach is to use specific tool designed for analyzing memory behaviour. One such is tool is Pympler, which can help you get more realistic idea about Python object sizes:

# pip install pympler
from pympler import asizeof
print(asizeof.asizeof([1, 2, 3, 4, 5]))
# 256

print(asizeof.asized([1, 2, 3, 4, 5], detail=1).format())
# [1, 2, 3, 4, 5] size=256 flat=96
#     1 size=32 flat=32
#     2 size=32 flat=32
#     3 size=32 flat=32
#     4 size=32 flat=32
#     5 size=32 flat=32

print(asizeof.asized([1, 2, [3, 4], "string"], detail=1).format())
# [1, 2, [3, 4], 'string'] size=344 flat=88
#     [3, 4] size=136 flat=72
#     'string' size=56 flat=56
#     1 size=32 flat=32
#     2 size=32 flat=32

Pympler provides asizeof module with function of same name which correctly reports size of the list as well all values it contains. Additionally, this module also has asized function, that can give us further size breakdown of individual components of the object.

Pympler has many more features though, including tracking class instances or identifying memory leaks. In case these are something that might be needed for your application, then I recommend checking out tutorials available in docs.

Saving Some RAM

Now that we know how to look for all kinds of potential memory issues, we need to find a way to fix them. Potentially, quickest and easiest solution can be switching to more memory-efficient data structures.

Python lists are one of the more memory-hungry options when it comes to storing arrays of values:

from memory_profiler import memory_usage

def allocate(size):
    some_var = [n for n in range(size)]

usage = memory_usage((allocate, (int(1e7),)))  # `1e7` is 10 to the power of 7
peak = max(usage)
print(f"Usage over time: {usage}")
# Usage over time: [38.85546875, 39.05859375, 204.33984375, 357.81640625, 39.71484375]
print(f"Peak usage: {peak}")
# Peak usage: 357.81640625

The simple function above (allocate) creates a Python list of numbers using the specified size. To measure how much memory it takes up we can use memory_profiler shown earlier which gives us amount of memory used in 0.2 second intervals during function execution. We can see that generating list of 10 million numbers requires more than 350MiB of memory. Well, that seems like a lot for a bunch of numbers. Can we do any better?

import array

def allocate(size):
    some_var = array.array('l', range(size))

usage = memory_usage((allocate, (int(1e7),)))
peak = max(usage)
print(f"Usage over time: {usage}")
# Usage over time: [39.71484375, 39.71484375, 55.34765625, 71.14453125, 86.54296875, 101.49609375, 39.73046875]
print(f"Peak usage: {peak}")
# Peak usage: 101.49609375

In this example we used Python's array module, which can store primitives, such as integers or characters. We can see that in this case memory usage peaked at just over 100MiB. That's a huge difference in comparison to list. You can further reduce memory usage by choosing appropriate precision:

import array
help(array)

#  ...
#  |  Arrays represent basic values and behave very much like lists, except
#  |  the type of objects stored in them is constrained. The type is specified
#  |  at object creation time by using a type code, which is a single character.
#  |  The following type codes are defined:
#  |
#  |      Type code   C Type             Minimum size in bytes
#  |      'b'         signed integer     1
#  |      'B'         unsigned integer   1
#  |      'u'         Unicode character  2 (see note)
#  |      'h'         signed integer     2
#  |      'H'         unsigned integer   2
#  |      'i'         signed integer     2
#  |      'I'         unsigned integer   2
#  |      'l'         signed integer     4
#  |      'L'         unsigned integer   4
#  |      'q'         signed integer     8 (see note)
#  |      'Q'         unsigned integer   8 (see note)
#  |      'f'         floating point     4
#  |      'd'         floating point     8

One major downside of using array as data container is that it doesn't support that many types.

If you plan to perform a lot of mathematical operations on the data, then you're probably better off using NumPy arrays instead:

import numpy as np

def allocate(size):
    some_var = np.arange(size)

usage = memory_usage((allocate, (int(1e7),)))
peak = max(usage)
print(f"Usage over time: {usage}")
# Usage over time: [52.0625, 52.25390625, ..., 97.28515625, 107.28515625, 115.28515625, 123.28515625, 52.0625]
print(f"Peak usage: {peak}")
# Peak usage: 123.28515625

# More type options with NumPy:
data = np.ones(int(1e7), np.complex128)
# Useful helper functions:
print(f"Size in bytes: {data.nbytes:,}, Size of array (value count): {data.size:,}")
# Size in bytes: 160,000,000, Size of array (value count): 10,000,000

We can see that NumPy arrays also perform pretty well when it comes to memory usage with peak array size of ~123MiB. That's a bit more than array but with NumPy you can take advantage of fast mathematical functions as well as types that are not supported by array such as complex numbers.

The above optimizations help with overall size of arrays of values, but we can make some improvements also to the size of the individual objects defined by Python classes. This can be done using __slots__ class attribute which is used to explicitly declare class properties. Declaring __slots__ on a class also has a nice side effect of denying creation of __dict__ and __weakref__ attributes:

from pympler import asizeof

class Normal:
    pass

class Smaller:
    __slots__ = ()

print(asizeof.asized(Normal(), detail=1).format())
# <__main__.Normal object at 0x7f3c46c9ce50> size=152 flat=48
#     __dict__ size=104 flat=104
#     __class__ size=0 flat=0

print(asizeof.asized(Smaller(), detail=1).format())
# <__main__.Smaller object at 0x7f3c4266f780> size=32 flat=32
#     __class__ size=0 flat=0

Here we can see how much smaller the Smaller class instance actually is. The absence of __dict__ removes whole 104 bytes from each instance which can save huge amount of memory when instantiating millions of values.

The above tips and tricks should be helpful in dealing with numeric values as well as class objects. What about strings, though? How you should store those generally depends on what you intend to do with them. If you're going to search through huge number of string values, then - as we've seen - using list is very bad idea. set might be a bit more appropriate if execution speed is important, but will probably consume even more RAM. Best option might be to use optimized data structure such as trie, especially for static data sets which you use for example for querying. As is common with Python, there's already a library for that, as well as for many other tree-like data structures, some of which you will find at https://github.com/pytries.

Not Using RAM At All

Easiest way to save RAM is to not use it in a first place. You obviously can't avoid using RAM completely, but you can avoid loading full data set at once and instead work with the data incrementally where possible. The simplest way to achieve this is by using generators which return a lazy iterator, which computes elements on demand rather than all at once.

Stronger tool that you can leverage is memory-mapped files, which allows us to load only parts of data from a file. Python's standard library provides mmap module for this, which can be used to create memory-mapped files which behave both like files and bytearrays. You can use them both with file operations like read, seek or write as well as string operations:

import mmap

with open("some-data.txt" "r") as file:
    with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) as m:
        print(f"Read using 'read' method: {m.read(15)}")
        # Read using 'read' method: b'Lorem ipsum dol'
        m.seek(0)  # Rewind to start
        print(f"Read using slice method: {m[:15]}")
        # Read using slice method: b'Lorem ipsum dol'

Loading/reading memory-mapped file is very simple. We first open the file for reading as we usually do. We then use file's file descriptor (file.fileno()) to create memory-mapped file from it. From there we can access its data both with file operations such as read or string operations like slicing.

Most of the time, you will be probably interested more reading the file as shown above, but it's also possible to write to the memory-mapped file:

import mmap
import re

with open("some-data.txt", "r+") as file:
    with mmap.mmap(file.fileno(), 0) as m:
        # Words starting with capital letter
        pattern = re.compile(rb'\b[A-Z].*?\b')

        for match in pattern.findall(m):
            print(match)
            # b'Lorem'
            # b'Morbi'
            # b'Nullam'
            # ...

        # Delete first 10 characters
        start = 0
        end = 10
        length = end - start
        size = len(m)
        new_size = size - length
        m.move(start, end, size - end)
        m.flush()
    file.truncate(new_size)

First difference in the code that you will notice is the change in access mode to r+, which denotes both reading and writing. To show that we can indeed perform both reading and writing operations, we first read from the file and then use RegEx to search for all the words that start with capital letter. After that we demonstrate deletion of data from the file. This is not as straightforward as reading and searching, because we need to adjust size of the file when we delete some of its contents. To do so, we use move(dest, src, count) method of mmap module which copies size - end bytes of the data from index end to index start, which in this case translates to deletion of first 10 bytes.

If you're doing computations in NumPy, then you might prefer its memmap features (docs) which is suitable for NumPy arrays stored in binary files.

Closing Thoughts

Optimizing applications is difficult problem in general. It also heavily depends on the task at hand as well as the type of data itself. In this article we looked at common ways to find memory usage issues and some options for fixing them. There are however many other approaches to reducing memory footprint of an application. This includes trading accuracy for storage space by using probabilistic data structures such as bloom filters or HyperLogLog. Another option is using tree-like data structures like DAWG or Marissa trie which are very efficient at storing string data.

Upcoming Python Features Brought to You by Python Enhancement Proposals

Martin Heinz — Mon, 14 Feb 2022 18:23:13 +0000

Before any new feature, change or improvement makes it into Python, there needs to be a Python Enhancement Proposal, also knows as PEP, outlining the proposed change. These PEPs are a great way of getting the freshest info about what might be included in the upcoming Python releases. So, in this article we will go over all the proposals that are going to bring some exciting new Python features in a near future!

Syntax Changes

All these proposals can be split into a couple of categories, first of them being syntax change proposals which are bound to bring interesting features.

First in this category is PEP 671 which proposes syntax for late-bound function argument defaults. What does that mean, though?

Well, functions in Python can take other functions as arguments. There's however no good way to set default value for such arguments. Usually None or sentinel value (global constant) is used as default which has disadvantages, including inability to use help(function) on the arguments. This PEP describes new syntax using => (param=>func()) notation to specify functions as default parameters.

# Current solution:
_SENTINEL = object()

def func(param=_SENTINEL):
    if param is _SENTINEL:
        # default_param holds expected default value
        param = default_param

# New solution:
def func(param=>default_param):
    ...

This change looks reasonable and useful to me, but I think we should be cautious of adding too many new syntax notations/changes. It's questionable whether small improvement like this one warrants yet another assignment operator.

Another syntax change proposal is PEP 654, which proposes except* as a new syntax for raising groups of exceptions. The rationale for this one is that Python interpreter can only propagate one exception at the time, but sometimes multiple unrelated exceptions need to be propagated as the stack unwinds. One such case is concurrent errors from asyncio from concurrent tasks or multiple different exceptions raised when performing retry logic, e.g., when connecting to some remote host.

try:
    some_file_operation()
except* OSError as eg:
    for e in eg.exceptions:
        print(type(e).__name__)

# FileNotFoundError
# FileExistsError
# IsADirectoryError
# PermissionError
# ...

This is a very simple example of using this new feature. If you take a look at the examples of handling Exceptions Groups in the PEP you will find a lot of wild ways to use this, including recursive matching and chaining.

Typing

Next category - which has been very heavily present in recent Python releases - is typing/type annotations.

Let's start with PEP 673 - which doesn't require extensive knowledge of Python's typing module (as is usually the case). Let's explain it with an example: Let's say you have a class Person with method set_name, which returns self - that being - instance of type Person. If you then create subclass Employee with same set_name method, you'd expect it to return instance of type Employee rather than Person. That's however not how type checking currently works - in Python 3.10 type checker infers return type in subclass to be that of base class. This PEP fixes that, by allowing us to use the "Self Type" with the following syntax that will help type checker to infer types correctly:

class Person:
    def set_name(self, name: str) -> Self:
        self.name = name
        return self

If you've run into this issue, then you can look forward to using this feature fairly soon, because this PEP is accepted and will be implemented as part of Python 3.11 release.

Another typing change is presented in PEP 675 which is titled Arbitrary Literal Strings.

Introduction of this PEP stems from the fact that it's currently not possible to specify that type of function argument can be an arbitrary literal string (only specific literal string can be used, e.g. Literal["foo"]). You might be wondering why is this even a problem and why would someone need to specify that parameter should be literal string and specifically not f-string (or other interpolated string). It's mostly a matter of security - requiring that parameter is literal helps avoid injections attacks, whether it's SQL/command injection or for example XSS. Some examples of these are shown in PEP's appendix. Having this implemented would help libraries like sqlite to provide warnings to users when string interpolation is used where it shouldn't be, so let's hope this one gets accepted soon.

Debugging

Next up are PEPs that help us debug our code a bit more efficiently. Starting off with PEP 669, titled Low Impact Monitoring for CPython.This PEP proposes to implement low cost monitoring for CPython that would not impact performance of Python programs when running debuggers or profilers. This is not something that will greatly impact end-users of Python, considering that there aren't big performance penalties when doing basic debugging. This can however be very useful in some niche cases, such as:

Debugging an issue that can only be reproduced in production environment while not impacting application performance.
Debugging race conditions where timing can affect whether the issue will occur or not.
Debugging while running a benchmark.

I personally probably won't benefit that much from this, but I'm sure people who are trying to improve performance of Python itself would definitely appreciate this change, as it would make it easier to debug and benchmark performance issues/improvements.

Next one is PEP 678 which suggests that __note__ attribute should be added to BaseException class. This attribute would be used to hold additional debugging information which could be displayed as part of traceback.

try:
    raise TypeError('Some error')
except Exception as e:
    e.__note__ = 'Extra information'
    raise

# Traceback (most recent call last):
#   File "<stdin>", line 2, in <module>
# TypeError: Some error
# Extra information

As shown in the example above, this is especially useful when re-raising exception. As the PEP describes, this would be also useful for libraries that have retry logic, adding extra information for each failed attempt. Similarly, testing libraries could use this to add more context to failed assertions, such as variable names and values.

Final debugging-related proposal is PEP 657 which wants to add additional data to every bytecode instruction of Python programs. This data can be used to generate better traceback information. It also suggests that API should be exposed which would allow other tools such as profilers or static analysis tools to make use of the data.

This might not sound that interesting, but it's actually - in my opinion - the most useful PEP presented here. The big benefit of this PEP will definitely be having nicer traceback information, such as:

Traceback (most recent call last):
  File <example.py>", line 5, in <module>
    value * (a - b * c)
             ~~^~~~~~~
TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'

Traceback (most recent call last):
  File <example.py>", line 10, in <module>
    some_dict['first']['second']['third']
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable

I think this is an amazing improvement to traceback readability and in general to debugging, and I'm really happy that this got implemented as part of Python 3.11, so we will get to use it very soon.

Quality of Life Changes

Final topic is dedicated to PEPs that bring certain "quality of life" improvements. One of those is PEP 680, which proposes to add support for parsing TOML format to Python's standard library.

TOML as a format is by default used by many Python tools, including build tools. This creates a bootstrapping problem for them. Additionally, many popular tools such as flake8 don't include TOML support citing its lack of support in standard library. This PEP proposes to add TOML support to standard library based on tomli which is already used by packages such as pip or pytest.

I personally like this proposal, and I think it makes sense to include libraries for common/popular formats in standard library, especially when they're so important to Python's tooling and ecosystem. The question is then, when will we see YAML support in Python standard library?

Last but not least, is PEP 661, which is related to so-called "sentinel values". There's no standard way of creating such values in Python. It's usually done with _something = object() (common idiom) as shown with PEP 671 earlier. This PEP proposes specification/implementation of sentinel values for standard library:

# Old:
_sentinel = object()

# New:
from sentinels import sentinel
some_name = sentinel("some_name")

Apart from the new solution being more readable, this can also help with type annotations as it will provide distinct type for all sentinels.

Closing Thoughts

From the proposals presented above, it's clear that a lot of great stuff is coming to Python. Not all these features will however make it into Python (at least not in their current state), so make sure to keep an eye on these proposals to see where they are going. To keep up-to-date with above PEPs as well as any new additions, you can take an occasional peek at the index or get notified about every new addition by subscribing to PEP RSS feed.

Additionally, I only included PEPs that propose some new feature/change to the language, there are however other ones that specify best practices, processes or oven Python's release schedule, so make sure to check the above mentioned index, if you're interested in those topics.

Creating Beautiful Tracebacks with Python's Exception Hooks

Martin Heinz — Wed, 02 Feb 2022 08:08:41 +0000

We all spend a good chuck of our time debugging, sifting through logs or reading tracebacks. Each of these can be difficult and time-consuming and in this article we will focus on making the last one - dealing with tracebacks and exceptions - as easy and efficient as possible.

To achieve this we will learn how to implement and use custom Exception Hooks that will remove all the noise from tracebacks, make them more readable and display just the information we need to troubleshoot our code and exceptions in Python. On top of that, we will also take a look at awesome Python libraries that provide ready to use exception hooks with beautiful tracebacks, which can be installed and used without any additional coding.

Exception Hooks

Whenever exception is raised and isn't handled by try/except block, a function assigned to sys.excepthook is called. This function - called Exception Hook - is then used to output any relevant information to standard output using the 3 arguments it receives - type, value and traceback.

Let's now look at a minimal example to see how this works:

import sys

def exception_hook(exc_type, exc_value, tb):
    print('Traceback:')
    filename = tb.tb_frame.f_code.co_filename
    name = tb.tb_frame.f_code.co_name
    line_no = tb.tb_lineno
    print(f"File {filename} line {line_no}, in {name}")

    # Exception type and value
    print(f"{exc_type.__name__}, Message: {exc_value}")

sys.excepthook = exception_hook

In the above example we leverage each of the arguments to provide basic traceback data in output. We use traceback (tb) object to access the traceback frame which contains data describing where the exception occurred - that is - filename (f_code.co_filename), function/module name (f_code.co_name) and line number (tb_lineno).

Apart from that, we also print information about exception itself using the exc_type and exc_value variables.

With this hook in place, we can invoke a function that raises some exception and we will receive the following output:

def do_stuff():
    # ... do something that raises exception
    raise ValueError("Some error message")

do_stuff()

# Traceback:
# File /home/some/path/exception_hooks.py line 22, in <module>
# ValueError, Message: Some error message

The above example provides some information about exception, but to get all the information needed for debugging, as well as a full picture of where and why the exception happened, we need to dig a bit deeper into the traceback object:

def exception_hook(exc_type, exc_value, tb):

    local_vars = {}
    while tb:
        filename = tb.tb_frame.f_code.co_filename
        name = tb.tb_frame.f_code.co_name
        line_no = tb.tb_lineno
        print(f"File {filename} line {line_no}, in {name}")

        local_vars = tb.tb_frame.f_locals
        tb = tb.tb_next
    print(f"Local variables in top frame: {local_vars}")

...

# File /home/some/path/exception_hooks.py line 41, in <module>
# File /home/some/path/exception_hooks.py line 7, in do_stuff
# Local variables in top frame: {'some_var': 'data'}

As you can see here, the traceback object (tb) is actually a linked list of all the exceptions that occurred - a stacktrace. This allows us to loop through it using tb_next and print information for each frame. On top of that, we can also use tb_frame.f_locals attribute to dump local variables to console, which can also aid in debugging.

Digging through the traceback object like we saw above works, but it's cumbersome and becomes quite unreadable very quickly. Better solution is to use traceback module instead, which provides lots of helper functions for extracting information about exceptions.

So, now that we know the basics, let's see how we can build our own exception hooks with some real, useful features...

Make Your Own

There are more things that we can do then just dump data on stdout. One of them would be logging the output to a file automatically:

LOG_FILE_PATH = "./some.log"
FILE = open(LOG_FILE_PATH, mode="w")

def exception_hook(exc_type, exc_value, tb):
    FILE.write("*** Exception: ***\n")
    traceback.print_exc(file=FILE)

    FILE.write("\n*** Traceback: ***\n")
    traceback.print_tb(tb, file=FILE)

# *** Exception: ***
# NoneType: None
# 
# *** Traceback: ***
#   File "/home/some/path/exception_hooks.py", line 82, in <module>
#     do_stuff()
#   File "/home/some/path/exception_hooks.py", line 7, in do_stuff
#     raise ValueError("Some error message")

This can be useful if you want to preserve information about uncaught exception for later debugging.

By default, uncaught exception will go to stderr, which might be undesirable if you have a logging setup in place and want the logger to take care of the error output. You could use following hook to allow logger to take care of these exceptions:

import logging
logging.basicConfig(
    level=logging.CRITICAL,
    format='[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s',
    datefmt='%H:%M:%S',
    stream=sys.stdout
)

def exception_hook(exc_type, exc_value, exc_traceback):
    logging.critical("Uncaught exception:", exc_info=(exc_type, exc_value, exc_traceback))


# [17:28:33] {/home/some/path/exception_hooks.py:117} CRITICAL - Uncaught exception:
# Traceback (most recent call last):
#   File "/home/some/path/exception_hooks.py", line 122, in <module>
#     do_stuff()
#   File "/home/some/path/exception_hooks.py", line 7, in do_stuff
#     raise ValueError("Some error message")
# ValueError: Some error message

The first thing that comes to mind when trying to improve the console output is making it pretty by giving it some colours highlighting the important bits:

# pip install colorama
from colorama import init, Fore
init(autoreset=True)  # Reset the color after every print

def exception_hook(exc_type, exc_value, tb):

    local_vars = {}
    while tb:
        filename = tb.tb_frame.f_code.co_filename
        name = tb.tb_frame.f_code.co_name
        line_no = tb.tb_lineno
        # Prepend desired color (e.g. RED) to line
        print(f"{Fore.RED}File {filename} line {line_no}, in {name}")

        local_vars = tb.tb_frame.f_locals
        tb = tb.tb_next
    print(f"{Fore.GREEN}Local variables in top frame: {local_vars}")

There's obviously much more you could do, for example printing local variables in each frame, or even lookup variables that were referenced on the line on which the exception occurred. Unsurprisingly, exception hooks for these use-cases already exist, so instead of dumping the code on you, I'd suggest you take a look at their source code from which you can draw some inspiration.

Finally, I want to include a word of caution, whenever you decide to install an exception hook, be aware that libraries can install their own hooks, so make sure you don't override those. In those cases you can instead catch the exception and use except block to output information you want to see, for example by using sys.exc_info().

Awesome Hooks in The Wild

Building your own exception hook can be a fun little exercise, but there are already quite a few cool ones out there. So, instead of reinventing a wheel, let's rather look at what we can just grab and start using immediately.

First and my favourite being Rich:

# https://rich.readthedocs.io/en/latest/traceback.html
# pip install rich
# python -m rich.traceback

from rich.traceback import install
install(show_locals=True)

do_stuff()  # Raises ValueError

The installation is super easy, all you need to do is install the library, import it and run install function which puts exception hook in place. If you want to just check out a sample output without writing Python code, then you can also use python -m rich.traceback.

Another popular option is better_exceptions. It also produces nice output, but requires a little more setup:

# https://github.com/Qix-/better-exceptions
# pip install better_exceptions
# export BETTER_EXCEPTIONS=1

import better_exceptions
better_exceptions.MAX_LENGTH = None
# Check if you TERM variable is set to `xterm`, if not set below variable - https://github.com/Qix-/better-exceptions/issues/8
better_exceptions.SUPPORTS_COLOR = True
better_exceptions.hook()

do_stuff()  # Raises ValueError

In addition to installing the library with pip we also need to set BETTER_EXCEPTIONS=1 environment variable to enable it. Next, we need the above Python code for setup. The most important part being the call to hook function which installs the exception hook. Additionally, we also set SUPPORTS_COLOR to True which might be necessary depending on the terminal you're using - more specifically - you will need this if your TERM variable is set to anything other than xterm.

Next up is pretty_errors library. This one is definitely the easiest one to configure, requiring just an import:

# https://github.com/onelivesleft/PrettyErrors/
# pip install pretty_errors

import pretty_errors
# `configure` can be omitted if you're satisfied with default settings
pretty_errors.configure(
    filename_display    = pretty_errors.FILENAME_EXTENDED,
    line_number_first   = True,
    display_link        = True,
    line_color          = pretty_errors.RED + '> ' + pretty_errors.default_config.line_color,
    code_color          = '  ' + pretty_errors.default_config.line_color,
    truncate_code       = True,
    display_locals      = True
)

do_stuff()

Apart from the mandatory import, the above snippet also shows optional configuration for the library. This is just a small sample of what you can configure to produce below output. The full list of config options can be found here.

Next one is a library whose output style will be familiar to everyone who uses Jupyter notebook. It's IPython's ultratb module which provides a couple of options for very pretty and readable exception and traceback error outputs:

# https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.ultratb.html
# pip install ipython
import IPython.core.ultratb

# Also ColorTB, FormattedTB, ListTB, SyntaxTB
sys.excepthook = IPython.core.ultratb.VerboseTB(color_scheme='Linux')  # Other colors: NoColor, LightBG, Neutral

do_stuff()

Last but not least is stackprinter library which produces concise output with all the debugging information you might need. Again, all you need to do to set it up is install the exception hook:

# https://github.com/cknd/stackprinter
# pip install stackprinter
import stackprinter
stackprinter.set_excepthook(style='darkbg2')

do_stuff()

Conclusion

In this article we learned how to write an exception hook, but I don't actually recommend writing and using your own hooks. Implementing one such hook could be a fun exercise, but probably not a worthwhile effort. You're better off using one of the awesome hooks presented above and calling it a day.

I do however, strongly recommend choosing one of them and installing it across all the projects you're working on, both for improved debugging, but also for consistency. The more you use one of these exception hooks, the more used you will become to its output and consequently, the more benefit you will get from using it.

With that said though, you should consider excluding custom exception hooks from your production build, as the prettified outputs might obscure some information which might be critical in certain scenarios. One such example would be missing file paths in some of the outputs above, which aids readability when debugging locally, but might make it harder to debug code running on remote system.

Building GitHub Apps with Golang

Martin Heinz — Mon, 17 Jan 2022 19:11:57 +0000

If you're using GitHub as your version control system of choice then GitHub Apps can be incredibly useful for many tasks including building CI/CD, managing repositories, querying statistical data and much more. In this article we will walk through the process of building such an app in Go including setting up the GitHub integration, authenticating with GitHub, listening to webhooks, querying GitHub API and more.

TL;DR: All the code used in this article is available at https://github.com/MartinHeinz/go-github-app.

Choosing Integration Type

Before we jump into building the app, we first need to decide which type of integration we want to use. GitHub provides 3 options - Personal Access Tokens, GitHub Apps and OAuth Apps. Each of these 3 have their pros and cons, so here are some basic things to consider:

Personal Access Token is the simplest form of authentication and is suitable if you only need to authenticate with GitHub as yourself. If you need to act on behalf of other users, then this won't be good enough
GitHub Apps are the preferred way of developing GitHub integrations. They can be installed by individual users as well as whole organizations. They can listen to events from GitHub via webhooks as well as access the API when needed. They're quite powerful, but even if you request all the permissions available, you won't be able to use them to perform all the actions that a user can.
OAuth Apps use OAuth2 to authenticate with GitHub on behalf of user. This means that they can perform any action that user can. This might seem like the best option, but the permissions don't provide the same granularity as GitHub Apps, and it's also more difficult to set up because of OAuth.

If you're not sure what to choose, then you can also take a look at diagram in docs which might help you decide. In this article we will use GitHub App as it's very versatile integration and best option for most use cases.

Setting Up

Before we start writing any code, we need to create and configure the GitHub App integration:

As a prerequisite, we need a tunnel which we will use to deliver GitHub webhooks from internet to our locally running application. You will need to install localtunnel tool with npm install -g localtunnel and start forwarding to your localhost using lt --port 8080.
Next we need to go to https://github.com/settings/apps/new to configure the integration. Fill the fields as follows:
- Homepage URL: Your localtunnel URL
- Webhook URL: https://<LOCALTUNNEL_URL>/api/v1/github/payload
- Webhook secret: any secret you want (and save it)
- Repository Permissions: Contents, Metadata (Read-only)
- Subscribe to events: Push, Release
After creating the app, you will be presented with the settings page of the integration. Take note of App ID, generate a private key and download it.
Next you will also need to install the app to use it with your GitHub account. Go to Install App tab and install it into your account.
We also need installation ID, which we can find by going to Advanced tab and clicking on latest delivery in the list, take a note of installation ID from request payload, it should be located in { "installation": { "id": <...>} }.

If you've got lost somewhere along the way, refer to the guide GitHub docs which shows where you can find each of the values.

With that done, we have the integration configured and all the important values saved. Before we start receiving events and making API requests we need to get the Go server up and running, so let's start coding!

Building the App

To build the Go application, we will use the template I prepared in https://github.com/MartinHeinz/go-github-app. This application is ready to be used as GitHub app and all that's missing in it, are a couple of variables which we saved during setup in previous section. The repository contains convenience script which you can use to populate all the values:

git clone git@github.com:MartinHeinz/go-github-app.git && cd go-github-app
./configure_project.sh \
    APP_ID="54321" \
    INSTALLATION_ID="987654321" \
    WEBHOOK_SECRET="verysecret" \
    KEY_PATH="./github_key.pem" \
    REGISTRY="ghcr.io/<GITHUB_USERNAME>/go-github-app"

The following sections will walk you through the code but if you're inpatient, then the app is good to go. You can use make build to build a binary of the application or make container to create a containerized version of it.

First part of the code we need to tackle is authentication. It's done using ghinstallation package as follows:

func InitGitHubClient() {
    tr := http.DefaultTransport
    itr, err := ghinstallation.NewKeyFromFile(tr, 12345, 123456789, "/config/github-app.pem")

    if err != nil {
        log.Fatal(err)
    }

    config.Config.GitHubClient = github.NewClient(&http.Client{Transport: itr})
}

This function, which is invoked from main.go during Gin server start-up, takes App ID, Installation ID and private key to create a GitHub client which is then stored in global config in config.Config.GitHubClient. We will use this client to talk to the GitHub API later.

Along with the GitHub client, we also need to set up server routes so that we can receive payloads:

func main() {
    // ...
    v1 := r.Group("/api/v1")
    {
        v1.POST("/github/payload", webhooks.ConsumeEvent)
        v1.GET("/github/pullrequests/:owner/:repo", apis.GetPullRequests)
        v1.GET("/github/pullrequests/:owner/:repo/:page", apis.GetPullRequestsPaginated)
    }
    utils.InitGitHubClient()
    r.Run(fmt.Sprintf(":%v", config.Config.ServerPort))
}

First of these is the payload path at http://.../api/v1/github/payload which we used during GitHub integration setup. This path is associated with webhooks.ConsumeEvent function which will receive all the events from GitHub.

For security reasons, the first thing the webhooks.ConsumeEvent function does is verify request signature to make sure that GitHub is really the service that generated the event:

func VerifySignature(payload []byte, signature string) bool {
    key := hmac.New(sha256.New, []byte(config.Config.GitHubWebhookSecret))
    key.Write([]byte(string(payload)))
    computedSignature := "sha256=" + hex.EncodeToString(key.Sum(nil))
    log.Printf("computed signature: %s", computedSignature)

    return computedSignature == signature
}

func ConsumeEvent(c *gin.Context) {
    payload, _ := ioutil.ReadAll(c.Request.Body)

    if !VerifySignature(payload, c.GetHeader("X-Hub-Signature-256")) {
        c.AbortWithStatus(http.StatusUnauthorized)
        log.Println("signatures don't match")
    }
    // ...

It performs the verification by computing a HMAC digest of payload using webhook secret as a key, which is then compared with the value in X-Hub-Signature-256 header of a request. If the signatures match then we can proceed to consuming the individual events:

func ConsumeEvent(c *gin.Context) {
    // ...
    event := c.GetHeader("X-GitHub-Event")

    for _, e := range Events {
        if string(e) == event {
            log.Printf("consuming event: %s", e)
            var p EventPayload
            json.Unmarshal(payload, &p)
            if err := Consumers[string(e)](p); err != nil {
                log.Printf("couldn't consume event %s, error: %+v", string(e), err)
                // We're responding to GitHub API, we really just want to say "OK" or "not OK"
                c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"reason": err})
            }
            log.Printf("consumed event: %s", e)
            c.AbortWithStatus(http.StatusNoContent)
            return
        }
    }
    log.Printf("Unsupported event: %s", event)
    c.AbortWithStatusJSON(http.StatusNotImplemented, gin.H{"reason": "Unsupported event: " + event})
}

In the above snippet we extract the event type from X-GitHub-Event header and iterate through a list of events that our app supports. In this case those are:

const (
    Install     Event = "installation"
    Ping        Event = "ping"
    Push        Event = "push"
    PullRequest Event = "pull_request"
)

var Events = []Event{
    Install,
    Ping,
    Push,
    PullRequest,
}

If the event name matches one of the options we proceed with loading the JSON payload into a EventPayload struct, which is defined in cmd/app/webhook/models.go. It's just a struct generated using https://mholt.github.io/json-to-go/ with unnecessary fields stripped.

That payload is then sent to function that handles the respective event type, which is one of the following:

var Consumers = map[string]func(EventPayload) error{
    string(Install):     consumeInstallEvent,
    string(Ping):        consumePingEvent,
    string(Push):        consumePushEvent,
    string(PullRequest): consumePullRequestEvent,
}

For example for push event one can do something like this:

func consumePushEvent(payload EventPayload) error {
    // Process event ...
    // Insert data into database ...
    log.Printf("Received push from %s, by user %s, on branch %s",
        payload.Repository.FullName,
        payload.Pusher.Name,
        payload.Ref)

    // Enumerating commits
    var commits []string
    for _, commit := range payload.Commits {
        commits = append(commits, commit.ID)
    }
    log.Printf("Pushed commits: %v", commits)

    return nil
}

That being in this case - checking the receiving repository and branch and enumerating the commits contained in this single push. This is the place where you could for example insert the data into database or send some notification regarding the event.

Now we have the code ready, but how do we test it? To do so, we will use the tunnel which you already should have running, assuming you followed the steps in previous sections.

Additionally, we also need to spin up the server, you can do that by running make container to build the containerized application, followed by make run which will start the container that listens on port 8080.

Now you can simply push to one of your repositories and you should see a similar output in the server logs:

[GIN] 2022/01/02 - 14:44:10 | 204 |     696.813µs |   123.82.234.90 | POST     "/api/v1/github/payload"
2022/01/02 14:44:10 Received push from MartinHeinz/some-repo, by user MartinHeinz, on branch refs/heads/master
2022/01/02 14:44:10 Pushed commits: [9024da76ec611e60a8dc833eaa6bca7b005bb029]
2022/01/02 14:44:10 consumed event: push

To avoid having to push dummy changes to repositories all the time, you can redeliver payloads from Advanced tab in your GitHub App configuration. On this tab you will find a list of previous requests, just choose one and hit the Redeliver button.

Making API Calls

GitHub apps are centered around webhooks to which you can subscribe and listen to, but you can also use any of the GitHub REST/GraphQL API endpoints assuming you requested the necessary permissions. Using API rather than push events is useful - for example - when creating files, analyzing bulk data or querying data which cannot be received from webhooks.

For demonstration of how to do so, we will retrieve pull requests of specified repository:

func GetPullRequests(c *gin.Context) {
    owner := c.Param("owner")
    repo := c.Param("repo")
    if pullRequests, resp, err := config.Config.GitHubClient.PullRequests.List(
        c, owner, repo, &github.PullRequestListOptions{
        State: "open",
    }); err != nil {
        log.Println(err)
        c.AbortWithStatus(resp.StatusCode)
    } else {
        var pullRequestTitles []string
        for _, pr := range pullRequests {
            pullRequestTitles = append(pullRequestTitles, *pr.Title)
        }
        c.JSON(http.StatusOK, gin.H{
            "pull_requests": pullRequestTitles,
        })
    }
}

This function takes 2 arguments - owner and repo - which get passed to PullRequests.List(...) function of GitHub client instance. Along with that, we also provide PullRequestListOptions struct to specify that we're only interested in pull requests with state set to open. We then iterate over returned PRs and accumulate all their titles which we return in response.

The above function resides on .../api/v1/github/pullrequests/:owner/:repo path as specified in main.go so we can query it like so:

curl http://localhost:8080/api/v1/github/pullrequests/octocat/hello-world | jq .

It might not be ideal to query API as shown above in situations where we expect a lot of data to be returned. In those cases we can utilize paging to avoid hitting rate limits. A function called GetPullRequestsPaginated that performs the same task as GetPullRequests with addition of page argument for specifying page size can be found in cmd/app/apis/github.go.

Writing Tests

So far we've been testing the app with localtunnel, which is nice for quick ad-hoc tests against live API, but it doesn't replace proper unit tests. To write unit tests for this app, we need to mock-out the API to avoid being dependent on the external service. To do so, we can use go-github-mock:

func TestGithubGetPullRequests(t *testing.T) {
    expectedTitles := []string{ "PR number one", "PR number three" }
    closedPullRequestTitle := "PR number two"
    mockedHTTPClient := mock.NewMockedHTTPClient(
        mock.WithRequestMatch(
            mock.GetReposPullsByOwnerByRepo,
            []github.PullRequest{
                {State: github.String("open"), Title: &expectedTitles[0]},
                {State: github.String("closed"), Title: &closedPullRequestTitle},
                {State: github.String("open"), Title: &expectedTitles[1]},
            },
        ),
    )
    client := github.NewClient(mockedHTTPClient)
    config.Config.GitHubClient = client

    gin.SetMode(gin.TestMode)
    res := httptest.NewRecorder()
    ctx, _ := gin.CreateTestContext(res)
    ctx.Params = []gin.Param{
        {Key: "owner", Value: "octocat"},
        {Key: "repo", Value: "hello-world"},
    }

    GetPullRequests(ctx)
    body, _ := ioutil.ReadAll(res.Body)

    assert.Equal(t, 200, res.Code)
    assert.Contains(t, string(body), expectedTitles[0])
    assert.NotContains(t, string(body), closedPullRequestTitle)
    assert.Contains(t, string(body), expectedTitles[1])
}

This test starts by defining mock client which will be used in place of normal GitHub client. We give it list of pull request which will be returned when PullRequests.List is called. We then create test context with arguments that we want to pass to the function under test, and we invoke the function. Finally, we read the response body and assert that only PRs with open state were returned.

For more tests, see the full source code which includes examples of tests for pagination as well as handling of errors coming from GitHub API.

When it comes to testing our webhook methods, we don't need to use a mock client, because we're dealing with basic API requests. Example of such tests including generic API testing setup can be found in cmd/app/webhooks/github_test.go.

Conclusion

In this article I tried to give you a quick tour of both GitHub apps, as well as the GitHub repository containing the sample Go GitHub project. In both cases, I didn't cover everything, the Go client package has much more to offer and to see all the actions you can perform with it, I recommend skimming through the docs index as well as looking at the source code itself where GitHub API links are listed along each function. For example, like the earlier shown PullRequests.List here.

As for the repository, there are couple more things you might want to take a look at, including Makefile targets, CI/CD or additional tests. If you have any feedback or suggestions, feel free to create an issue or just star it if it was helpful to you. 🙂

DEV Community: Martin Heinz

Advanced Features of Kubernetes' Horizontal Pod Autoscaler

Setup

Basic Autoscaling

Custom Metrics

External Metrics

HPAScaleToZero

HPAContainerMetrics

LogarithmicScaleDown

Closing Thoughts

Data and System Visualization Tools That Will Boost Your Productivity

JSON

Regular Expressions

YAML

SQL

Git

Docker

Kubernetes

Closing Thoughts

Stop Messing with Kubernetes Finalizers

Finalizers

What Could Go Wrong?

Finalizers in The Wild

Building Your Own

Conclusion

Automate All the Boring Kubernetes Operations with Python

Playground

Authentication

Raw Requests

Python Client

Handy Examples

Conclusion

End-to-End Monitoring with Grafana Cloud with Minimal Effort

Infrastructure Provider

Metrics

Dashboards

Synthetics

Alerts and Notifications

Logs

Traces

Closing Thoughts

Goodbye, Google Analytics - Why and How You Should Leave The Platform

Why?

How?

Alternatives

Closing Thoughts

Python f-strings Are More Powerful Than You Might Think

Date and Time Formatting

Variable Names and Debugging

__repr__ and __str__

Superior Performance

Full Power of Formatting Spec

Nested F-Strings

Conditionals Formatting

Lambda Expressions

Closing Thoughts

Ultimate CI Pipeline for All of Your Python Projects

Quick Start

The Basics

Code Quality

Package

Security

Notify

Closing Thoughts

Optimizing Memory Usage of Python Applications

Why Bother, Anyway?

Find Bottlenecks

Saving Some RAM

Not Using RAM At All

Closing Thoughts

Upcoming Python Features Brought to You by Python Enhancement Proposals

Syntax Changes

Typing

Debugging

Quality of Life Changes

Closing Thoughts

Creating Beautiful Tracebacks with Python's Exception Hooks

Exception Hooks

Make Your Own

Awesome Hooks in The Wild

`repr` and `str`