DEV Community

Cover image for Backing up Prometheus using TSDB Snapshots : Kubernetes/OpenShift
karan singh
karan singh

Posted on

4 2

Backing up Prometheus using TSDB Snapshots : Kubernetes/OpenShift

These are my quick-and-dirty brain-dump notes to myself on how to backup prometheus database running on k8s or OpenShift

  • Get Token for API Authentication and Prometheus API Route URL
    oc whoami -t
    oc get route -n openshift-monitoring | grep -i prometheus
Enter fullscreen mode Exit fullscreen mode
  • Run sample curl request
    curl -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/query?query=ALERTS'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/query?query=ALERTS') | python -m json.tool
Enter fullscreen mode Exit fullscreen mode
  • Create TSDB Snapshot
    curl -X ‘POST’ -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot') | python -m json.tool
Enter fullscreen mode Exit fullscreen mode
  • You might get an error, so you first need to enable Admin APIs
    {
     “error”: “Admin APIs are disabled”,
     “message”: “Admin APIs are disabled”,
     “code”: 14
    }
Enter fullscreen mode Exit fullscreen mode
  • Enable AdminAPI
    oc -n openshift-monitoring patch prometheus k8s \
     — type merge — patch ‘{“spec”:{“enableAdminAPI”:true}}’
Enter fullscreen mode Exit fullscreen mode
  • Verify Admin API is enabled
    oc describe po prometheus-k8s-1 | grep -i admin
     — web.enable-admin-api
Enter fullscreen mode Exit fullscreen mode
  • Hit TSDB snapshot API to take snapshot
    curl -X ‘POST’ -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot')
     | python -m json.tool
Enter fullscreen mode Exit fullscreen mode
    {
     “name”: “20210512T162601Z-33415dbd315ae6af”
    }
Enter fullscreen mode Exit fullscreen mode
  • Find the snapshot and copy it locally

  • The default folder is /prometheus/snapshots/ but you can find the data folder by finding the --storage.tsdb.path config in your deployment.

    curl -X ‘POST’ -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/admin/tsdb/snapshot'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/admin/tsdb/snapshot') | python -m json.tool
Enter fullscreen mode Exit fullscreen mode
  • List the the snapshot directory
    oc -n openshift-monitoring exec -it prometheus-k8s-0 -c prometheus — /bin/sh -c “ls /prometheus/snapshots/20210512T162601Z-33415dbd315ae6af”
Enter fullscreen mode Exit fullscreen mode
  • Copy the Snapshot from prometheus container to local machine
    oc project openshift-monitoring
    oc rsync prometheus-k8s-0:/prometheus/snapshots/ /home/prometheus
Enter fullscreen mode Exit fullscreen mode

And this is how to back Prometheus Snapshot to a local machine

Image of AssemblyAI tool

Challenge Submission: SpeechCraft - AI-Powered Speech Analysis for Better Communication

SpeechCraft is an advanced real-time speech analytics platform that transforms spoken words into actionable insights. Using cutting-edge AI technology from AssemblyAI, it provides instant transcription while analyzing multiple dimensions of speech performance.

Read full post

Top comments (0)

Heroku

This site is powered by Heroku

Heroku was created by developers, for developers. Get started today and find out why Heroku has been the platform of choice for brands like DEV for over a decade.

Sign Up

👋 Kindness is contagious

Immerse yourself in a wealth of knowledge with this piece, supported by the inclusive DEV Community—every developer, no matter where they are in their journey, is invited to contribute to our collective wisdom.

A simple “thank you” goes a long way—express your gratitude below in the comments!

Gathering insights enriches our journey on DEV and fortifies our community ties. Did you find this article valuable? Taking a moment to thank the author can have a significant impact.

Okay