CI said green. Production said otherwise: building an AI change flight recorder on SigNoz

Siddhant — Sat, 25 Jul 2026 23:57:10 +0000

A one-line config change passed all eight of my CI checks. It was reviewed and
shipped. Production p95 on the affected endpoint then rose 7.3×.

Nothing was broken in a way CI could see. The value parsed, the tests passed, the
container built. The failure lived in the gap between "the pipeline is green" and
"the change is safe."

This post is about closing that gap with SigNoz, and about the five mistakes that
collapsed the first time I ran it end to end. If you are wiring CI data into an
observability backend, the mistakes are the part worth your time.

The one-line regression

Here is the change, on a real commit
2fa6e28:

   "data_source": {
     "max_open_conns": 20,
-    "max_idle_conns": 5
+    "max_idle_conns": 5,
+    "conn_max_lifetime": 1000000
   },

It reads like ordinary connection-pool tuning. But the service is
Blnk, written in Go, and a JSON number
decoded into time.Duration is nanoseconds. 1000000 is not sixteen
minutes. It is one millisecond. The pool threw away every PostgreSQL
connection almost as soon as it opened it.

Under measured traffic on /balances, p95 moved from 1.44 ms to 10.45 ms. A
later revert measured 2.1 ms.

I did not write this bug into a toy service. Blnk is a third-party Apache-2.0
ledger, checksum-verified at v0.15.1, which knows nothing about my tooling. A
regression your own demo was built to detect proves little.

The part SigNoz does not do out of the box

GitHub Actions exports no OpenTelemetry. A workflow run is REST metadata —
timestamps, conclusions, job names — not spans. So a CI result and a production
trace live in two systems and cannot be asked one question.

So I reconstruct each run as a real trace: one root span per run, one child per
job, with real start and end times, exported over OTLP as greenlight-ci. Once
CI, deployments and request traffic are all spans in one backend, "which run
approved this version, and what did it do to latency?" is one query.

The unit of comparison is the immutable deployed version, not wall-clock
time. Every deployment reports its commit SHA as service.version:

service.name = 'blnk-loan-workload'
  AND service.version = '<commit sha>'
  AND deployment.environment.name = 'hackathon-demo'
  AND http.route = '/balances'

That makes rollbacks and overlapping deploys unambiguous. A baseline captured
hours earlier still refers to exactly one version.

Which session wrote this line?

An agent wrote that regression. That is ordinary now, and it changes the question
you ask after an incident: not just what changed, but which session produced
it, and what was it told to do?

Claude Code emits OpenTelemetry, and with CLAUDE_CODE_PROPAGATE_TRACEPARENT=1
it sets TRACEPARENT for the commands it runs — so a prepare-commit-msg hook
can write that context into the commit:

AI-Traceparent: 00-86b83e0039724d54b250693de0e7cba7-95cf03c6c3e1413b-01

On sync, the worker resolves that exact trace and span against
service.name = 'claude-code'. If it resolves, the receipt reads
AI link: verified. If a trailer names a span SigNoz does not hold, it reads
failed — distinct from missing, which means no trailer at all. A commit that
claimed a session and could not back it up is a stronger statement than one that
never claimed anything.

Enabling OTEL_LOG_USER_PROMPTS took this further: the receipt now shows that
session's prompts, so the instruction, the commit and the latency graph are one
chain. Tool arguments and output stay off — they carry file contents and command
output — and CI fails the build if either is enabled.

Two caveats. Anything typed into an instrumented session is recorded, so it is no
place for a credential. And the trailer proves a commit was made from a
session, not that a model wrote the code. Session provenance, not authorship.

Five things that broke when I ran it for real

1. My load generator was measuring itself. It took a duration flag but never
paced requests. Hundreds of calls finished instantly, Blnk's rate limiter
rejected a chunk, and my "healthy" baseline showed an error rate the test tool
had manufactured. If your baseline looks unhealthy before you change anything,
suspect the harness first.

2. My latency threshold silently exempted fast endpoints. The policy required
p95 to rise 1.5× and by at least 250 ms — a number I picked because it is
roughly what a user notices. On a 1.44 ms route, that means reaching 251 ms. A
174× regression would not have qualified. I replaced it with a 2 ms floor, a
measurement-resolution guard rather than a perception one.

3. My first demo measured two things at once. I injected a PostgreSQL outage
inside the candidate's measured window, so the verdict fired on an error rate
unrelated to the config change. The fix was splitting the clean chain and the
dependency-failure scenario into two explicitly labelled runs.

4. A trace is not a session. I read a session's prompts by filtering on
trace_id, which seemed obviously correct. But trace context propagates into
nested sessions: an agent started from inside another inherits its parent's
TRACEPARENT and joins the same trace. One trace ended up holding three
sessions, and the receipt was ready to show an unrelated throwaway prompt as the
explanation for a database commit. Attribution now resolves the trailer's span to
its owning session.id first.

5. My validator could never pass where I told people to run it. It checked
that OTEL_* exports were set, and the docs said to run it from inside the coding
session — but Claude Code strips OTEL_* from the environment it gives the
commands it runs. It reported a broken pipeline while spans arrived the whole
time.

That third one is why every receipt carries this, and why it is load-bearing:

Deployment correlation is evidence of temporal and version association, not
proof that every observed failure was caused by the commit.

How each SigNoz signal earns its place

Traces decide the verdict. Two Query Builder v5 queries per window: one
returns count, p90 and p95, the other counts spans with has_error = true over
the same scope. One detail cost me an hour — SigNoz answers an empty window with
a zero count and null percentiles, and Number(null) is 0. Coerced naively,
"no traffic" reads as "zero latency".

Dashboards make versions comparable. The panel that matters groups p95 by
service.version, so baseline, candidate and recovery are three series and the
step between them is the deployment.

Metrics describe decisions, which traces cannot. A verdict is not a request
and queue depth is a state, not an event, so both are custom instruments —
greenlight.regression.verdicts by status, plus gauges observed on the export
interval so an idle-but-healthy system still reports.

Alerts follow the service, not a version. Two rules — p95, and a true error
rate computed as A/B*100 over errored spans and all spans. Deliberately not
pinned to service.version: a version-pinned alert can only describe a version
that existed when you wrote the rule.

Logs carry the commit. API and worker logs ship over OTLP with trace context,
and any job about a change carries commit_sha — an investigator arrives holding
a commit, not a queue job ID.

MCP answers the same question agent-natively. I ask the SigNoz MCP server to
compare two versions over streamable HTTP, with no direct-query fallback. Over a
wide 15-hour window it reported 1.58 ms vs 9.39 ms, corroborating the receipt's
narrower 1.44 → 10.45 ms. Error rates differ (0% vs 9.13%) because the wide
window also contains the dependency-failure rehearsal — same version, different
windows, different correct answers.

Try it

Node 24, Docker Compose v2, SigNoz Foundry v0.2.16.

git clone https://github.com/sid12701/greenlight-ai-change-flight-recorder
cd greenlight-ai-change-flight-recorder
npm ci && cp .env.demo.example .env.demo
npm run demo:up

It pauses once, on purpose: SigNoz does not expose an API key through automation,
so you create a service-account key in the UI and re-run.

My last clean run passed lint, type-checking, every build, 245 tests
(13 skipped), 24 receipt links, six image-digest checks and three MCP trace
resolutions.

What I would tell myself at the start

Verify end to end before you believe any number. Every failure above survived unit tests. Two were invisible until a real coding session ran.
Measure one thing at a time, or you have measured nothing you can attribute.
Say what you did not observe. My alert rules fired and resolved, but I never watched SigNoz deliver the webhook end to end, so I do not claim it.

Boundaries: the service map is empty because this workload has no cross-service
parent/child spans, and the demo is local rather than hosted.

Closing

Observability cannot prove a commit caused an incident, and I would not trust a
tool that claimed otherwise. What it can do is make the evidence resolvable,
version-scoped, and honest enough that a human — or an agent — starts from facts
instead of guesswork. When the thing writing your code also emits telemetry, that
evidence reaches back to the instruction that started it.

The code is on GitHub,
MIT licensed.

AI assistance disclosure: Codex/ChatGPT and Claude Code were used for
planning, implementation, review and submission preparation, as allowed by the
hackathon rules. All commits
are reviewed and authored under my own Git identity. See
PROVENANCE.md.

DEV Community: Siddhant