Building a Safe-by-Design HTTP Telemetry Crate in Rust

Sylvain TIASSOU — Sun, 15 Nov 2026 17:18:59 +0000

Incidents are easier to investigate when you can see what happened inside your HTTP service: which route failed, how long it took, and what the request looked like.

But logging request data can create new problems. Passwords and tokens may end up in logs. Large payloads can consume memory. And if telemetry export waits on a slow or unavailable backend, it can affect the request path.

To address these tradeoffs, I built safe-http-telemetry, an open-source Rust crate for Axum and Tower applications.

Its design goal is to make HTTP telemetry useful while keeping capture, resource use, and export behavior under the application’s control.
Three design priorities

Redact sensitive data before queueing When an application captures request data, it needs to consider credentials such as passwords, bearer tokens, and API keys.

safe-http-telemetry applies capture limits and redaction before events enter the export queue. It supports configured sensitive-field matching in query parameters, headers, and supported structured body formats. JSON objects are traversed recursively, with limits on depth, visited nodes, and string size.

Body capture is optional. It requires the body-capture feature and a route configured for the Detailed profile. Applications can also add field names specific to their own domain.

Redaction is a safeguard, not a guarantee that every secret will be identified. I recommend starting with the Minimal profile, enabling richer capture only on reviewed routes, and limiting access to the telemetry backend.

Keep the request path independent from Loki The middleware submits events to a bounded queue without waiting for a network export. Background workers can consume and batch events for asynchronous delivery to Grafana Loki.

If the queue fills up, the default DropNewest policy rejects the incoming event and updates operational counters. This keeps queue memory bounded and avoids waiting for Loki to become available. The application can monitor these counters and configure limits to suit its workload.

The event flow looks like this:

HTTP request
     │
     ▼
Axum / Tower middleware
     │
     ▼
Route policy → bounded capture → redaction
     │
     ▼
Bounded event queue ── full? ──► drop newest + update counters
     │
     ▼
Background exporter workers
     │
     ▼
Grafana Loki

The queue and exporter have separate responsibilities: the middleware prepares events, while the consumer handles delivery and shutdown.

Make geolocation an explicit local option For applications that need it, the optional geolocation feature can look up country information using a local MMDB database. The database is supplied separately, loaded during setup, and is not downloaded by the crate. Lookups do not call an external geolocation API.

The cache has configurable capacity and time-to-live limits. By default, the event includes country code and country name; the application can choose other supported fields. Geolocation depends on resolving the client IP correctly, so forwarded headers should only be trusted from configured proxy networks.
A small Axum integration surface
After creating a bounded queue and route policies, attach the layer to your router:

let telemetry_layer =
    axum_telemetry_layer(queue.clone(), policies, MiddlewareConfig::default())?;

let app = Router::new()
    .route("/hello/{name}", get(handler))
    .layer(telemetry_layer);

With Axum, register routes before applying the layer so the middleware can record normalized route templates. The crate leaves server setup, queue consumption, and shutdown sequencing to the application.
What the benchmarks show
I used Criterion to measure middleware overhead with tower::ServiceExt::oneshot. The recorded run compared a service without telemetry to three capture profiles:

Baseline: 2.38 µs
Minimal: 27.31 µs
Standard: 60.16 µs
Detailed: 71.58 µs These are estimates from a documented run on one machine, not universal performance guarantees. The benchmark does not measure network throughput, tail latency under load, or sustained exporter outages. The geolocation case in that run did not supply a client IP, so it did not measure an actual database lookup.

For a real service, benchmark with representative payloads, concurrency, and capture policies.
Testing malformed and unexpected input
The repository includes property-based tests with proptest for configuration, policy resolution, and queue capacity. It also includes cargo-fuzz targets for query, header, and JSON redaction inputs.

These checks help exercise boundary conditions and malformed data. They complement application-specific testing, especially when adding custom sensitive field names or enabling detailed body capture.
What I learned
Building this crate reinforced that observability is not just about collecting more logs. It also involves data minimization, backpressure, resource limits, and graceful behavior when a backend is unavailable.

safe-http-telemetry is available on crates.io. If you build with Axum or Tower, I’d welcome feedback on the integration, documentation, performance, and any edge cases you find.

Crate: crates.io/crates/safe-http-telemetry
Repository: github.com/sylvainTIASSOU/safe-http-telemetry Bug reports, feature suggestions, and contributions are very welcome. 🦀

DEV Community: Sylvain TIASSOU

Building a Safe-by-Design HTTP Telemetry Crate in Rust