DEV Community: Alessandro Vozza

Stateless, Secretless Multi-cluster Monitoring in Azure Kubernetes Service with Thanos, Prometheus and Azure Managed Grafana

Alessandro Vozza — Mon, 25 Jul 2022 09:15:52 +0000

Introduction

Observability is paramount to every distributed system and it's becoming increasingly complicated in a cloud native world where we might deploy multiple ephemeral clusters and we want to keep their metrics beyond their lifecycle span.

This article aims at cloud native engineers that face the challenge of observing multiple Azure Kubernetes Clusters (AKS) and need a flexible, stateless solution, leveraging available and cost-effective blob storage for long term retention of metrics, one which does not require injecting static secrets to access the storage (as it leverage the native Azure Managed Identities associated with the cluster).

This solution builds upon well-established Cloud Native Computing Foundation (CNCF) open source projects like Thanos and Prometheus,together with a new managed services, Azure Managed Grafana, recently released in public preview. It allows for ephemeral clusters to still have updated metrics without the 2-hours local storage of metrics in the classic deployment of Thanos sidecar to Prometheus.

This article was inspired by several sources, most importantly this two articles: Using Azure Kubernetes Service with Grafana and Prometheus and Store Prometheus Metrics with Thanos, Azure Storage and Azure Kubernetes Service on Microsoft Techcommunity blog.

Prerequisites

An 1.23 or 1.24 AKS cluster with either a user-managed identity assigned to the kubelet identity or system-assigned identity
Ability to assign roles on Azure resources (User Access Administrator role)
A storage account
(Recommended) A public DNS zone in Azure
Azure CLI
Helm CLI

Architecture

We will deploy all components of Thanos and Prometheus in a single cluster, but since they are couple only via the ingress they don't need to be co-located.

Cluster-wide services

For Thanos receive and query components to be available outside the cluster and secured with TLS, we will need ingress-nginx and cert-manager. For ingress, deploy the Helm chart using the following command, to account for this issue with AKS clusters >1.23:

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \
  --set controller.service.externalTrafficPolicy=Local \
  --namespace ingress-nginx --create-namespace

Notice the extra annotations and the externalTrafficPolicy set to Local.

Next, we need cert-manager to automatically provision SSL certificates from Let's Encrypt; we will just need a valid email address for the ClusterIssuer:

helm upgrade -i cert-manager \
  --namespace cert-manager --create-namespace \
  --set installCRDs=true \
  --set ingressShim.defaultIssuerName=letsencrypt-prod \
  --set ingressShim.defaultIssuerKind=ClusterIssuer \
  --repo https://charts.jetstack.io cert-manager

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: email@email.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

Last but not least, we will add a DNS record for our ingress Loadbalancer IP, so it will be seamless to get public FQDNs for our endpoints for Thanos receive and Thanos Query.

az network dns record-set a add-record  -n "*.thanos" -g dns -z cookingwithazure.com \
--ipv4-address $(kubectl get svc ingress-nginx-controller -n ingress-nginx -o jsonpath="{.status.loadBalancer.ingress[0].ip}")

Note how we use kubectl with jsonpath type output to get the ingress public IP. We can now leverage the wildcard FQDN *.thanos.cookingwithazure.com in our ingresses and cert-manager will be able to obtain the relative certificate seamlessly.

Storage account preparation

Because we do not want to store any secret or service principal in-cluster, we will leverage the Managed Identities assigned to the cluster and assign the relevant Azure Roles to the storage account.

Once you have created or identified the storage account to use and created a container within it, to store the Thanos metrics, assign the roles using the azure cli; first, determine the clientID of the managed identity:

clientid=$(az aks show -g <rg> -n <cluster_name> -o json --query identityProfile.kubeletidentity.clientId)

Now, assign the role of Reader and Data Access to the Storage account (you need this so the cloud controller can generate access keys for the containers) and the Storage Blob Data Contributor role to the container only (there's no need to give this permission at the storage account level, because it will enable writing to every container, which we don't need. Always remember to apply the principles of least privileges!)

az role assignment create --role "Reader and data access" --assignee $clientid --scope /subscriptions/<subID>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<account_name>

az role assignment create --role "Storage Blob Data Contributor" --assignee $clientid --scope /subscriptions/<subID>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<account_name>/containers/<container_name>

Create basic auth credentials

Ok, we kinda cheated in the title: you do need one credential at least for this setup, and it's the one to access the Prometheus API exposed by Thanos from Azure Managed Grafana. We will use the same credentials (but feel free to generate a different one) to push metrics from Prometheus to Thanos using remote-write via the ingress controller. You'll need a strong password stored into a file called pass locally:

htpasswd -c  -i auth thanos < pass

#Create the namespaces
kubectl create ns thanos
kubectl create ns prometheus

#for Thanos Query and Receive
kubectl create secret generic -n thanos basic-auth --from-file=auth

#for Prometheus remote write
kubectl create secret generic -n prometheus remotewrite-secret \
--from-literal=user=thanos \
--from-literal=password=$(cat pass)

We now have the secrets in place for the ingresses and for deploying Prometheus.

Deploying Thanos

We will use the Bitnami chart to deploy the Thanos components we need.

helm upgrade -i thanos -n monitoring --create-namespace --values thanos-values.yaml bitnami/thanos

Let's go thru the relevant sections of the values file:

objstoreConfig: |-
  type: AZURE
  config:
    storage_account: "thanostore"
    container: "thanostore"
    endpoint: "blob.core.windows.net"
    max_retries: 0
    user_assigned_id: "5c424851-e907-4cb0-acb5-3ea42fc56082"

(replace the user_assigned_id with the object id of your kubeletIdentity, for more information about AKS identities, check out this article) This section instructs the Thanos Store Gateway and Compactor to use an Azure Blob store, and to use the kubelet identity to access it. Next, we enable the ruler and the query components:

ruler:
  enabled: true

query:
  enabled: true
...
queryFrontend:
  enabled: true

We also enable autoscaling for the stateless query components (the query and the query-frontend; the latter helps aggregating read queries), and we enable simple authentication for the Query frontend service using ingress-nginx annotations:

queryFrontend:
...
  ingress:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/auth-secret: basic-auth
      nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - thanos'
    hostname: query.thanos.cookingwithazure.com
    ingressClassName: nginx
    tls: true

The annotation references the basic-auth secret we created before from the htpasswd credentials. Note that the same annotations are also under the receive section, as we're using the exact same secret for pushing metrics into Thanos (although with a different hostname).

Prometheus remote-write

Until full support for Agent mode lands in the Prometheus operator (follow this issue), we can use the remote write feature to ship every metrics instantly to a remote endpoint, in our case represented by the Thanos Query Frontend ingress. Let's start by deploying Prometheus using the kube-prometheus-stack helm chart:

helm  upgrade -i -n prometheus promremotewrite -f prom-remotewrite.yaml prometheus-community/kube-prometheus-stack

Let's go thru the values file to explain the options we need to enable remote-write:

prometheus:
  enabled: true
  prometheusSpec:
    externalLabels:
      datacenter: westeu
      cluster: playground

This enables Prometheus and attaches two extra labels to every metrics, so it becomes easier to filter data coming from multiple sources/clusters later in Grafana.

    remoteWrite:
    - url: "https://receive.thanos.cookingwithazure.com/api/v1/receive"
      name: Thanos
      basicAuth:
        username:
          name: remotewrite-secret
          key: user
        password:
          name: remotewrite-secret
          key: password

This section points to the remote endpoint (secured via SSL using Let's Encrypt certificates, thus trusted by the certificate store on the AKS nodes; if you use a non-trusted certificate, refer to the TLSConfig section of the PrometheusSpec API). Note how the credentials to access the remote endpoint are coming from the secret created beforehand and stored in the prometheus namespace.

Note here that although Prometheus is deployed in the same cluster as Thanos for simplicity, it sends the metrics to the ingress FQDN, thus it's trivial to extend this setup to multiple, remote clusters and collect their metrics into a single, centralized Thanos receive collector (and a single blob storage), with all metrics correctly tagged and identifiable.

Observing the stack with Azure Managed Grafana

Azure Managed Grafana(AME) is a new offering in the toolset of observability tools in Azure, and it's based on the popular open source dashboarding system Grafana. Beside out of the box integration with Azure, AME is a fully functional Grafana deployment that can be used to monitor and graph different sources, including Thanos and Prometheus. To start, head to the Azure Portal and deploy AME; then, get the endpoint from the Overview tab and connect to your AME instance.

Add a new source of type Prometheus and basic authentication (the same we created before):

Congratulations! We can now visualize the data flowing from Prometheus, we only need a dashboard to properly display the data. Go to (on the left side navigation bar) Dashboards-> Browse and click on Import; import the "Kubernetes / Views / Global" (ID: 15757) into your Grafana and you'll be able to see the metrics from the cluster:

The imported dashboard has no filter for cluster or region, thus will show all cluster metrics aggregated. We will show in a future post how to add a variable to a Grafana dashboard to properly select and filter cluster views.

Future work

This setup allows for autoscaling of receiver and query frontend as horizontal pod autoscalers are deployed and associated with the Thanos components. For even greater scalability and metrics isolation, Thanos can be deployed multiple times (each associated with different storage accounts as needed) each with a different ingress to separate at the source the metrics (thus appearing as separate sources in Grafana, which can then be displayed in the same dashboard, selecting the appropriate source for each graph and query).

Managing multiple clusters with ArgoCD in Azure/k3s secured w/ Traefik&Let’s Encrypt

Alessandro Vozza — Fri, 16 Oct 2020 15:51:03 +0000

Easily deploy ArgoCD in Azure to manage multiple clusters in a GitOps way.

Code available at https://github.com/ams0/argocd-azure-k3s-traefik

I heard about ArgoCD many times (recently from my friends at Fullstaq here) but never came around to kick its tires until now. If you don’t know, ArgoCD is a platform for declarative continuous deployment of Kubernetes applications, and it’s becoming quickly an exceedingly popular (and now it’s an incubated CNCF project) choice to deploy and manage applications at scale on multiple clusters.

Since I want to use it for deploying to a cluster, my plan is to have an ArgoCD instance outside my clusters that can manage them independently from the clusters’ lifecycle; hence, I devise this method of deploying ArgoCD into a VM in azure running the lightweight distribution of Kubernetes from Rancher Labs, k3s (deployed using the k3d helper tool) and exposed via the Traefik ingress controller and secured with Let’s Encrypt certificates. Let’s get to it!

Start by cloning the repo and entering it:

git clone [*https://github.com/ams0/argocd-azure-k3s-traefik](https://github.com/ams0/argocd-azure-k3s-traefik)
cd argocd-azure-k3s-traefik*

Now some prerequisites:

Azure CLI
Azure subscription (already logged in)
kubectl and jq installed

Let’s create the infrastructure (one VM with some extra ports open to access the Kubernetes APIs and 80/443 to expose our application):

./deploy.sh <rg> <dns_name> <location> <size>

dns_name should be unique in the region of choice. After a couple of minutes, you’ll have the config file for the k3s cluster (of one VM, with two virtual nodes inside as docker containers).

You may have noticed that I skipped the installation of k3s’ built-in Traefik ingress; that’s because it still packs the 1.7 branch and I want to use the newer 2.x branch that introduced the IngressRoute CRD (you can follow the progress on this issue). Let’s now install traefik with this script (you’ll need to pass your email for the Let’sEncrypt certificate authority):

./install-traefik <email>

Finally, install ArgoCD passing the same dns_name/region and a password of your choice:

./install-argo.sh <dns_name> <region> <password>

That’s it! The script will patch argocd-server to run over http (SSL termination is done by Traefik) and will patch the secret with the bcrypt-encoded version of your password. Navigate to *https://dns_name.region.cloudapp.azure.com* and login in ArgoCD. You can also download the CLI and login with:

argocd login --username admin \
--password Password \
dns_name.region.cloudapp.azure.com

Finally you can add one or more clusters to be managed by argo with (provided you already have the kubeconfig file available, for example using the azure cli to retrieve it):

az aks get-credentials -g rg -n *cluster_name* -f kubeconfig
argocd cluster add --kubeconfig  ./kubeconfig manageme

Note that the last option on the above command line must match the context inside the kubeconfig configuration.

Now, let’s deploy some apps!

I’m a big fan of the Helm-controller project and I wanted to use it with ArgoCD. In a nutshell, the controller lets you create objects of kind HelmRelease *in your cluster (representing Helm releases) and manages the lifecycle of those helm releases programmatically (create/update/destroy). So in the *manifests/ folder in my repository, you’ll find the templates to deploy an helm-controller that will in turn deploy helmreleases also present in the same folder.

Fork the repository I provided at the top of this post and head over to the Argo UI tocreate a new app (call it to your liking, and choose the default project) pointing to the your fork and to the manifests/ path.

Importantly, make sure that directory recurse is on. You can also create an object of type “Application” inside your Argo/k3s cluster to achieve the same result:

The app will start syncing right away, installing the Helm operator first and then an nginx ingress controller and you’ll see the tree of resources being created.

That’s it! Now every change to the github repo will be reflected in your cluster.

In conclusion, you might be tempted to ask: is this production ready? Absolutely not, there are still some aspects of the deployment I want to improve (AAD authentication for Argo, for instance, and persistence of data as well). However, it’s a quickstart with ArgoCD in Azure and it will help me learn more in the future. I hope you enjoy it too!

Originally published on Medium

Hi, I'm Alessandro Vozza

Alessandro Vozza — Thu, 01 Jun 2017 14:22:06 +0000

I have been coding for null years.

You can find me on GitHub as ams0

I live in Amsterdam (The Netherlands).

I work for Microsoft.

I mostly program in these languages: is Kubernetes a language? .

I am currently learning more about node.

Nice to meet you.