DEV Community

Cover image for Backing up Prometheus using TSDB Snapshots : Kubernetes/OpenShift
karan singh
karan singh

Posted on

4 2

Backing up Prometheus using TSDB Snapshots : Kubernetes/OpenShift

These are my quick-and-dirty brain-dump notes to myself on how to backup prometheus database running on k8s or OpenShift

  • Get Token for API Authentication and Prometheus API Route URL
    oc whoami -t
    oc get route -n openshift-monitoring | grep -i prometheus
Enter fullscreen mode Exit fullscreen mode
  • Run sample curl request
    curl -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/query?query=ALERTS'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/query?query=ALERTS') | python -m json.tool
Enter fullscreen mode Exit fullscreen mode
  • Create TSDB Snapshot
    curl -X ‘POST’ -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot') | python -m json.tool
Enter fullscreen mode Exit fullscreen mode
  • You might get an error, so you first need to enable Admin APIs
    {
     “error”: “Admin APIs are disabled”,
     “message”: “Admin APIs are disabled”,
     “code”: 14
    }
Enter fullscreen mode Exit fullscreen mode
  • Enable AdminAPI
    oc -n openshift-monitoring patch prometheus k8s \
     — type merge — patch ‘{“spec”:{“enableAdminAPI”:true}}’
Enter fullscreen mode Exit fullscreen mode
  • Verify Admin API is enabled
    oc describe po prometheus-k8s-1 | grep -i admin
     — web.enable-admin-api
Enter fullscreen mode Exit fullscreen mode
  • Hit TSDB snapshot API to take snapshot
    curl -X ‘POST’ -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot')
     | python -m json.tool
Enter fullscreen mode Exit fullscreen mode
    {
     “name”: “20210512T162601Z-33415dbd315ae6af”
    }
Enter fullscreen mode Exit fullscreen mode
  • Find the snapshot and copy it locally

  • The default folder is /prometheus/snapshots/ but you can find the data folder by finding the --storage.tsdb.path config in your deployment.

    curl -X ‘POST’ -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/admin/tsdb/snapshot'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/admin/tsdb/snapshot') | python -m json.tool
Enter fullscreen mode Exit fullscreen mode
  • List the the snapshot directory
    oc -n openshift-monitoring exec -it prometheus-k8s-0 -c prometheus — /bin/sh -c “ls /prometheus/snapshots/20210512T162601Z-33415dbd315ae6af”
Enter fullscreen mode Exit fullscreen mode
  • Copy the Snapshot from prometheus container to local machine
    oc project openshift-monitoring
    oc rsync prometheus-k8s-0:/prometheus/snapshots/ /home/prometheus
Enter fullscreen mode Exit fullscreen mode

And this is how to back Prometheus Snapshot to a local machine

Postmark Image

Speedy emails, satisfied customers

Are delayed transactional emails costing you user satisfaction? Postmark delivers your emails almost instantly, keeping your customers happy and connected.

Sign up

Top comments (0)

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay