An Alloy user had one header configured on pyroscope.scrape, an X-Api-Key. About two hours in, profiling stopped. Their packet capture showed a single GET carrying roughly 1,000 copies of that header, and the target answering 431 Request Header Fields Too Large.
The bug was in prometheus/common, the library behind http_config in Prometheus, Alertmanager and the exporters. The reporter had already traced it to cloneRequest. I wrote the upstream fix, #982, which shipped in v0.72.1.
Reproducing it
One header in the config, the same request sent four times:
$ go run . # prometheus/common v0.71.0
server saw X-Api-Key x1
server saw X-Api-Key x2
server saw X-Api-Key x3
server saw X-Api-Key x4
$ go run . # prometheus/common v0.72.1
server saw X-Api-Key x1
server saw X-Api-Key x1
server saw X-Api-Key x1
server saw X-Api-Key x1
The copy that copied nothing
r2 := new(http.Request)
*r2 = *r
// Deep copy of the Header.
maps.Copy(r.Header, r2.Header)
The comment says deep copy. It isn't one. *r2 = *r leaves both requests pointing at the same header map, and maps.Copy copies that map into itself. Anything the library adds afterwards lands in the caller's request. http_headers uses Header.Add, so a request that gets reused picks up one more copy per send.
Checking your binaries
$ go version -m ./alloy | grep prometheus/common
dep github.com/prometheus/common v0.71.0 h1:...
v0.69.0 to v0.72.0 are affected. v0.72.1 or later is fixed. You only hit it with an affected version, an http_headers block, and code that sends the same request object twice.
Prometheus 3.15 pins the same broken version and never shows the bug, because an otelhttp wrapper clones each request first. The full post has the exposure table for five tools and the regression test that went in with the fix.
Top comments (0)