Pushing Prometheus metrics with pushgateway

Preface

This post is generally about pushing metrics in pushgateway, however, I will warn you and admit right away that the text will contain an example - an anti-pattern of pushing metrics, since using pushgateway is recommended in the case when the service does not work constantly (or the service/task being started does not have any interface at all), and therefore it is better for prometheus not to constantly knock on closed doors and not do unnecessary work.

Introduction

So, pushgateway is a service where you can push metrics when the standard prometheus pull-model is not applicable (in the preface, I generally described how such a situation can arise). After the metrics have entered the pushgateway, prometheus already picks them up from there, and this implies several restrictions related to pushing metrics, for example, the absence of the up metric, since it is generated by prometheus itself for the polled app instance, and in this case, it is only the pushgateway.

p.s. Although, if we talk about the up metric, then it is not needed if you use pushgateway in a recomended way.

Bake Prometheus for poll pushgateway

Let's say we have a compose like this with prometheus and pushgateway:

# ....(тут какие-нибудь графаны и т.д.)   
prometheus:  
    restart: always  
    image: bitnami/prometheus:latest  
    links:  
        - pushgateway  
    volumes:  
        - ./.prom.yml:/opt/bitnami/prometheus/conf/prometheus.yml  

pushgateway:  
    restart: always  
    image: bitnami/pushgateway:latest  
    ports:  
        - 9091:9091  

In this case, prom.yml should look something like this to collect data from pushgateway:

global: null
scrape_interval: 5s
scrape_timeout: 2s
evaluation_interval: 15s

scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets:
          - 'pushgateway:9091'

Everything is quite clear here, they added only honor_lables, which, if briefly resolves conflicts of label names, that is, for example, if your service has a metric with the "X" label and pushgateway has a label "X", then with honor_lables = false you will have label "X" with pushgateway and "exported_X" from your service that pushed metrics to pushgateway, and if true, only your service labels will be displayed (again, if there is a conflict).

p.s. Pushgateway security - docs recommends by default, for example, to use basic_auth.

Push metrics

I could give a beautiful example that corresponds to normal practice, however, I thought and decided that they hadn't given me any disrespect for a long time, therefore there will be an example of pushing metrics due to the fact that the service_discovery setting is absent (in production, of course, it is impossible).

So, let's say we have workers Faust there are a lot of them and they are not in the cluster (there is no swarm, no k8s), there are also no consul and other ways in which prometheus is able, they just run quietly in docker-compose and multiply with the scale argument.

More horror and sedition, in addition to pushing metrics, can be done by raging ports, for example, like this:

ports:  
- "9100-9200:6066"  

And write targets with the whole set of ports into the prometheus config.

Let's continue through the code. For metrics from workers, you can use an already ready library. All we will do in this case is write a small timer in order to send metrics using the standard push_to_gateway function.

async def push_metrics():  
    def auth_handler(url, method, timeout, headers, data):  
        return basic_auth_handler(url, method, timeout, headers, data, PUSHGATEWAY_USERNAME, PUSHGATEWAY_PASSWORD)  
    push_to_gateway(PUSHGATEWAY_URI, job=f"{WORKERS_APP_NAME}-{ENV}", registry=registry_metrics, handler=auth_handler)  

@app.timer(interval=PUSH_METRICS_INTERVAL)  
async def push_metrics_cron():  
    await push_metrics()  

As you can see, everything is quite simple here - we specify the job name (with a pool of metrics - this is done in the prometheus config), substitute the handler for authentication and specify the registry from which the metrics will be pushed. Well, that's all, we launch and when we open the pushgateway of the web-panel, we see that the metrics have been loaded at an interval, then the previously configured prometheus will take them from there.

Afterword

I decided to write a note, since I encountered a similar one in my work, I will say right away that the method from the example will not go to production, however, as using pushgateway in the absence of a service discovery, for testing it can come off.