DEV Community

Oleksandr Kuryzhev
Oleksandr Kuryzhev

Posted on Originally published at kuryzhev.cloud

What changed in Prometheus 3.16 for production use

Originally published on kuryzhev.cloud


The scenario: a new minor release and a server nobody wants to page on

Your Prometheus server has been quiet for months, scraping a few hundred targets and evaluating alert rules that other teams depend on. Then Prometheus 3.16 lands, released on 2026-11-04 according to the official release page. Prometheus ships a new minor roughly every six weeks, so the question is not whether to upgrade but how to do it without surprises in alerting or storage.

This is why it matters now. Grafana 13.3 and a stream of exporter updates are landing in the same weeks, and a monitoring server that falls many minors behind makes every later upgrade larger. The release notes are the authority on what is in 3.16.0. This guide gives you a repeatable procedure to find the changes that touch your setup and prove the new binary behaves before it owns production.

Short answer: read the [CHANGE] entries for 3.16.0, run the new promtool against your real config and rules, then canary 3.16.0 beside the old server with its own data directory and no alerting or remote write. Promote it only if the metrics in the verify section stay clean. This applies if you already run Prometheus 3.x. On 2.x, do the 3.0 migration first.

Prerequisites

You need a few things in place before touching a binary:

  • A server already on a 3.x release. The 3.0 breaking changes (UTF-8 metric names, stricter scrape content-type handling, left-open range selectors, removed Alertmanager v1 API) are covered in the official migration guide. Do not combine that jump with a minor bump.
  • Your configuration and rule files in version control, so you can diff and roll back.
  • Docker (or the release tarball) to run promtool from the 3.16.0 build.
  • A way to snapshot the data volume: an LVM, EBS or ZFS snapshot, or a stopped-server copy of the TSDB directory.
  • A list of the command-line flags and --enable-feature values your service unit currently passes.

When not to bother yet: if you run a long-term-support line for stability, you may deliberately skip non-LTS minors. Check the project's release schedule on prometheus.io to see which line is marked LTS. The cost of staying put is that you batch more changes into each later upgrade.

Step 1: Extract the breaking and feature entries for 3.16.0

Prometheus keeps a CHANGELOG.md where every bullet is tagged [CHANGE], [FEATURE], [ENHANCEMENT] or [BUGFIX]. The [CHANGE] lines are the ones that can alter behavior you rely on. This pulls just those for one release:

# Print only CHANGE and FEATURE bullets from the 3.16.0 section
curl -sL https://raw.githubusercontent.com/prometheus/prometheus/main/CHANGELOG.md \
  | awk '/^## 3\.16\.0/{f=1;print;next} /^## /{f=0} f' \
  | grep -E '^\* \[(CHANGE|FEATURE)\]'

# What flags and feature flags does your current server actually use?
systemctl cat prometheus | grep -E 'enable-feature|--storage|--web|--enable'

Compare the two outputs line by line. Put anything that mentions a flag you pass, a config key you set, or a metric name your dashboards query on a checklist. I am deliberately not listing the 3.16 entries here. Read them from the source, because a summary written from memory is how people miss the one line that matters.

Watch out for: feature flags. A flag you enabled to get an experimental behavior may be promoted to default, renamed, or become a no-op. The changelog entry is where that is announced.

Step 2: Validate config and rules with the new promtool

The cheapest safety net is running the target version's own promtool against your real files. It catches removed config keys and rule syntax problems before any process restarts. Adjust the paths to your layout:

# promtool from the 3.16.0 image, your files mounted read-only
docker run --rm --entrypoint promtool \
  -v /etc/prometheus:/etc/prometheus:ro \
  prom/prometheus:v3.16.0 \
  check config /etc/prometheus/prometheus.yml

# Rule files get a dedicated check too
docker run --rm --entrypoint promtool \
  -v /etc/prometheus:/etc/prometheus:ro \
  prom/prometheus:v3.16.0 \
  check rules /etc/prometheus/rules/*.yml

A clean run reports SUCCESS for the config and for each rule file, with a count of rules found. Any FAILED line names the file and the problem. check config also opens the files your config references, such as rule files, file_sd files and TLS certificates. Mount those paths into the container as well, or you will see false failures. Fix real problems in Git. Then re-run against the old version's promtool too, so the same files are valid on both sides of a rollback.

Watch out for: a passing check config does not prove your scrapes still work. Scrape behavior, such as how content types are handled, only shows at runtime, which is what the canary is for.

Step 3: Run a canary beside the old server

Start 3.16.0 on a different port with a fresh data directory. Never point two servers at the same TSDB, since the data directory is locked by one process. Strip anything with external side effects from the canary's config copy: the alerting: block and any remote_write: entries. Otherwise you get duplicate pages and duplicate samples in long-term storage.

services:
  prometheus-canary:
    image: prom/prometheus:v3.16.0
    command:
      - --config.file=/etc/prometheus/prometheus.canary.yml  # copy without alerting/remote_write
      - --storage.tsdb.path=/prometheus                      # canary's own named volume, never shared
      - --storage.tsdb.retention.time=2d                     # short retention, it is disposable
      - --web.listen-address=:9091
      # re-add your existing --enable-feature flags only if the changelog still supports them
    volumes:
      - /etc/prometheus:/etc/prometheus:ro
      - canary-data:/prometheus
    ports:
      - "9091:9091"
volumes:
  canary-data: {}

Mount the volume at the image's own /prometheus path. The image runs as an unprivileged user, and Docker copies that directory's ownership into a new named volume. A mount point at a path that does not exist in the image is created owned by root, and the TSDB fails to open with a permission error. If port 9091 is already taken on the host, for example by a Pushgateway, pick another port.

Let the canary run through at least one full cycle of your slowest scrape interval, plus a rule group evaluation or two. A day is better if you have daily batch jobs that expose metrics.

The cost is real but small: roughly the memory and CPU of a second server for the canary period. On a very large instance, that may mean scraping only a subset of jobs with a trimmed config.

Step 4: Promote with a snapshot and a rollback path

Once the canary is clean, take a snapshot of the production data volume, then swap the binary or image tag. If you run the server under systemd or Kubernetes, change only the version in this step. Put unrelated tuning in a separate change so a regression has one suspect.

# Reload only picks up config, not a new binary, so restart for the upgrade
sudo systemctl restart prometheus

# Returns 200 once WAL replay has finished and the server is ready (503 before that)
curl -s -o /dev/null -w '%{http_code}\n' localhost:9090/-/ready

# Confirm the new version is the one running
curl -s localhost:9090/api/v1/status/buildinfo

Confirm that the config loaded with the prometheus_config_last_reload_successful metric in the verify section. Expect a restart to replay the write-ahead log, which takes longer as the head block grows.

Alerts with for: clauses restore their pending state from the ALERTS_FOR_STATE series stored in the TSDB. That restore is governed by --rules.alert.for-outage-tolerance and --rules.alert.for-grace-period. After a short restart, most pending alerts resume rather than reset, but a longer outage can still delay firing. Schedule the restart outside deploy windows and let the replay finish before judging health.

Rollback: keep the previous binary or image tag and the snapshot. Whether a given minor can safely read data written by a newer one is stated in the release notes, so check before relying on a downgrade. Restoring the snapshot is the fallback that does not depend on that.

Verify and test

Query the server itself. These are standard Prometheus self-metrics. Run them on the canary first, then on production after promotion:

# Right version running?
prometheus_build_info{version="3.16.0"}

# Config loaded (expect 1)
prometheus_config_last_reload_successful

# Targets down (compare to the old server; should match)
count(up == 0)

# Rule failures (expect no increase)
increase(prometheus_rule_evaluation_failures_total[30m])

# Dropped alert notifications (production only; the canary has no Alertmanager)
increase(prometheus_notifications_dropped_total[30m])

# Series count should be in the same range as before
prometheus_tsdb_head_series

Also compare a few important dashboard queries between the two servers in Grafana by adding the canary as a second data source. Identical values, within scrape-timing jitter, are what you want. A target that is up on the old server but down on the canary points to a scrape-behavior change, so look in the changelog for the matching entry. After the first compaction cycle, check prometheus_tsdb_compactions_failed_total and the server logs for errors.

Official reference for flags, configuration and metrics: prometheus.io documentation. More operations guides live at kuryzhev.cloud.

Closing thoughts

Treat Prometheus 3.16 like any minor release of the system your pager depends on. Read the tagged changelog lines, let the new promtool judge your real files, and let a side-effect-free canary prove scrape and rule behavior before the production server restarts. The whole procedure costs one extra container and a day of patience, and it turns a surprise into a diff you reviewed. Keep the snapshot until the new version has survived a full compaction cycle and a normal week of alerts.

Related

Top comments (0)