Grafana Tempo 3.1: Query-Based Trace Redaction Explained

Originally published on DevToolHub.

Grafana Tempo 3.1 shipped on 2026-09-30 with a feature that matters the moment something goes wrong, not when everything's fine: redaction by query. Before this release, scrubbing a leaked secret out of your traces meant enumerating trace IDs one at a time. Now you can hand Tempo a TraceQL query and it finds and removes every matching trace. Here's the exact command, verified against Grafana's own docs, plus the rolling-upgrade gotcha that can make a "successful" redaction silently incomplete.

What's Actually New in Grafana Tempo 3.1

Per Grafana's release notes, Tempo 3.1's headline changes are:

• Redaction by query — remove traces matching a TraceQL query instead of enumerating trace IDs.
• Kafka ingestion improvements — authenticated, TLS-encrypted Kafka connections (SASL SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, AWS_MSK_IAM), rack-aware fetching via client_rack to cut cross-zone transfer costs, and ingest.kafka.producer_compression for Azure Event Hubs' required gzip compression.
• TraceQL metrics extrapolation and arithmetic — extrapolate rates from head-sampled traces so metrics reflect real traffic volume, and combine aggregations with arithmetic to compute ratios.
• vParquet5 becomes the default block format. Existing vParquet4 blocks stay readable with no migration needed; you can pin the old format with storage.trace.block.version: vParquet4 if you need to.

This article focuses on redaction, since it's the feature most teams will actually need to reach for under pressure — and the one where getting the details wrong has real consequences. If you're building out observability more broadly, see our guide to real-time log monitoring for the logs side of the same problem.

The Real Incident: A Secret Leaked Into a Span Attribute

The scenario this feature exists for: an API key, a password, or a customer PII field ends up in a span attribute — logged by an overly verbose instrumentation library, a debug flag left on, or a bug in a custom span processor. It's now sitting in your trace storage, queryable by anyone with Explore access, until you do something about it.

Before Tempo 3.1, your options were either live with it until the retention period expired, or find every offending trace ID manually and redact them one by one. Query-based redaction skips that — useful whether you're running Tempo standalone or as part of a full Kubernetes observability stack.

Grafana Tempo 3.1 Redaction: The Exact Command

Per Tempo's tempo-cli documentation, the redact command has two modes. Trace-ID mode (the old way, still supported, up to 1000 IDs per request):

tempo-cli redact --tenant=<TENANT_ID> --trace-id=<TRACE_ID> [--trace-id=<TRACE_ID> ...] <scheduler-address>

Query mode (the new way):

tempo-cli redact --tenant=<TENANT_ID> --query=<TRACEQL_QUERY> --start=<START> --end=<END> <scheduler-address>

For the leaked-API-key scenario, assuming the key landed in a span attribute called http.request.header.authorization:

tempo-cli redact \
  --tenant=prod \
  --query='{ span.http.request.header.authorization = "sk-live-abc123" }' \
  --start=now-7d \
  --end=now \
  --dry-run \
  tempo-scheduler.prod.svc:9095

Run with --dry-run first — it evaluates the query and reports a match count without rewriting anything. Once the count looks right, drop --dry-run to actually redact.

A real constraint worth knowing before you write the query: Tempo's docs specify the TraceQL support here is narrower than a normal TraceQL search — redaction "supports only single spanset filters with = comparisons on resource.* or span.* attributes, joined by && or ||." You can't use the full TraceQL grammar (no duration comparisons, no nested spanset logic) — just flat equality checks on resource or span attributes.

What Redaction Actually Does — and Doesn't Do

This is the part generic release-note summaries skip: redaction doesn't strip one field out of a trace. Per the docs, the command "rewrites all blocks for a tenant that contain specific trace IDs. The traces are dropped from the new blocks." The entire matching trace is removed from object storage, not just the offending attribute. If you need to keep the rest of the trace and only scrub one field, this feature doesn't do that — it's all-or-nothing at the trace level.

Tempo's own safety warnings are worth repeating verbatim because they change how you should actually run this in production:

1. "Redaction rewrites blocks in object storage and can't be undone." There's no soft-delete, no recovery window.
2. Stop ingesting the sensitive data at its source first — fix the instrumentation bug before you clean up its output, or you're redacting a moving target.
3. Wait roughly 10 minutes for current blocks to flush to storage before running the redaction, so you're not racing data still in memory.
4. For query-based redaction, verify your match set in Grafana Explore first, then confirm with --dry-run before dropping it.
5. A single pass is never fully complete on a live system. Traces still held by block-builders (not yet flushed) aren't covered by a redaction run — you may need a second pass.

The Rolling-Upgrade Gotcha

This is the finding that doesn't show up if you only read the headline feature list. Per Grafana's own 3.1 release notes:

"Don't submit a windowed redaction until every scheduler and worker is on Tempo 3.1. An older worker ignores the window and redacts every query match in the block, with no error."

Translate that: if you're mid-rollout — some Tempo workers upgraded to 3.1, some still on 3.0 — and you submit a redaction with --start/--end to limit it to a narrow time window, any worker still running the old version ignores your time window entirely and redacts every trace matching the query across the whole block, with no error or warning telling you it happened. You could intend to scrub one afternoon's worth of traces and instead silently wipe weeks of data, discovering it only when someone asks where their traces went.

The practical rule: finish the Tempo 3.1 rollout across every scheduler and worker before you run your first windowed redaction. If you're not certain every component is upgraded, run trace-ID-based redaction instead — it doesn't have this failure mode. It's the same discipline covered in our broader list of DevOps tools worth having in your incident-response toolkit before you need them.

Frequently Asked Questions

Q: Does Tempo 3.1's redaction remove just the sensitive field, or the whole trace?
A: The whole trace. Per Tempo's docs, matching traces are dropped entirely from the rewritten blocks — there's no partial-field redaction.

Q: Can I undo a redaction if I made a mistake?
A: No. Tempo's docs state explicitly that redaction "rewrites blocks in object storage and can't be undone." Always run --dry-run first and verify the match count.

Q: What TraceQL syntax does the --query flag actually support?
A: A narrower subset than normal TraceQL search: single spanset filters using = comparisons on resource.* or span.* attributes, combined with && or || only. No duration filters, no nested spansets.

Q: Is it safe to use --start/--end windowed redaction during a rolling upgrade?
A: No — per Grafana's own warning, a worker still on an older Tempo version ignores the time window and redacts every match in the block with no error. Wait until every scheduler and worker is on 3.1 first.

Quick Summary:

• Tempo 3.1 (GA'd 2026-09-30) adds redaction by TraceQL query, alongside Kafka auth/TLS improvements, TraceQL metrics arithmetic, and vParquet5 as the new default block format
• The redact command: tempo-cli redact --tenant=<id> --query=<traceql> --start=<t> --end=<t> --dry-run <scheduler-address>
• Redaction removes the entire matching trace from storage, not just one attribute, and can't be undone
• Query syntax is limited to flat = comparisons on resource.*/span.* attributes joined by &&/||
• Don't run windowed (--start/--end) redaction mid-rollout — an old worker silently ignores the window and redacts the whole block

Grafana Tempo 3.1's redaction feature is worth having ready before you need it — check your Tempo version across every scheduler and worker now, not during the incident where you actually need to use it.