Prometheus Observability Platform: Handling multiple regions

When we have multiple regions such as the EU and US, we need to have a long-term storage solution running in both of those. If we want to combine the resources into a single query we need to use a query layer that can query both endpoints. One such component we can use is Promxy.

Promxy uses the same PromQL syntax as Prometheus and we can define server groups with multiple endpoints. In our case, we would define our EU and US long-term storage endpoints under one server group. We can then use the single Promxy endpoint to query both the EU and the US.

Demo

This example assumes that you have completed the following steps, as the components from those are needed:

• Prometheus Observability Platform: Prometheus
• Prometheus Observability Platform: Long-term storage

We can use the Helm chart offered in the Promxy repository to deploy a proxy to our Kubernetes cluster.

First we clone the repository, because the Helm chart is not published to a public registry:

git clone https://github.com/jacksontj/promxy.git

Then we navigate to the folder containing the Helm chart:

cd promxy/deploy/k8s/helm-charts/promxy/

We set up Promxy with the following server_groups:

server_groups:
  - static_configs:
    - targets:
      - vmcluster-victoria-metrics-cluster-vmselect.victoriametrics.svc.cluster.local:8481
      labels:
        region: eu
    scheme: http
    path_prefix: /select/0/prometheus

I have converted this to json so we can pass it to the helm chart:

{"server_groups":[{"static_configs":[{"targets":["vmcluster-victoria-metrics-cluster-vmselect.victoriametrics.svc.cluster.local:8481"],"labels":{"region":"eu"}}],"scheme":"http","path_prefix":"/select/0/prometheus"}]}

To install Promxy from the local Helm chart we use the following:

helm install promxy . --create-namespace --namespace promxy --set 'image.tag=latest' --set-json 'config.promxy={"server_groups":[{"static_configs":[{"targets":["vmcluster-victoria-metrics-cluster-vmselect.victoriametrics.svc.cluster.local:8481"],"labels":{"region":"eu"}}],"scheme":"http","path_prefix":"/select/0/prometheus"}]}'

We can now port-forward the Promxy service and access the web UI from http://localhost:9090:

kubectl port-forward -n promxy services/promxy 9090:8082

Here we can perform the same query for the kube_pod_container_status_restarts_total to verify that Promxy is able to reach the VictoriaMetrics data:

If we have more regions than just the US, we can add them under the server_groups and query multiple VictoriaMetrics instances from a single Promxy source.

We have now achieved setting up Promxy with server_groups to use for querying VictoriaMetrics instances.

Next part: Prometheus Observability Platform: Application metrics