In April I trained a small model to resolve enterprise IT tickets. At 4 AM the night before the Meta x PyTorch hackathon deadline, it started writing about Star Wars. The loss had been flat at exactly zero for thirty minutes. In GRPO that means every completion in a group is earning identical reward, and the cheapest way to do that is one exploit, run eight times per batch. My parsimony reward had an inverted branch. The model used it to write film history instead of fixing servers.
Here is the embarrassing part. I caught it by reading raw completions by hand, dumped to disk by a global Python list I had bolted into a reward function that same night. That list was my observability stack. I needed to see what the model was writing so badly that I hand rolled the worst telemetry pipeline imaginable at 4 AM.
Two nights ago I did what I should have done then: re-graded that training run through a live, instrumented evaluation pipeline, with self-hosted SigNoz on the other end. It surfaced a bug I did not know I had, and not the one I went looking for. In the middle of a run I had called successful, my model spent ten straight steps being punished for writing better.
What I actually ran
The system: Qwen2.5-3B trained with GRPO for 200 steps, 8 completions per step, six reward functions grading each one (format, ROUGE-L quality, citation grounding, calibration, parsimony, repetition). 1,600 graded completions, logged to parquet during training. I had not heard of SigNoz before this hackathon and assumed it was an agent monitoring tool. It is a general observability platform, and its own LLM observability guide lists evaluation and fine-tuning monitoring as a core pillar. A reward pipeline is an evaluation system.
To be precise: no gradients ran. I re-executed the six reward functions, the real April code, on the stored April prompts and completions, with OpenTelemetry wrapping every call. WandB is the experiment log, aggregated curves per step. SigNoz became the flight recorder, every individual completion queryable and alertable by any attribute I could invent. It will not replace WandB for gradients or checkpoints, and I would not try.
The build
foundryctl cast stood the stack up on a base M1 with 5GB handed to Docker, from this casting.yaml:
apiVersion: v1alpha1
kind: Installation
metadata:
name: signoz
spec:
deployment: { flavor: compose, mode: docker }
mcp: { spec: { enabled: true } }
Before instrumenting anything, a fidelity gate: re-grade one parquet against April's logged scores. Max drift: 2.44e-08. Same code, bit for bit. (The tiny drift that did exist was a float32 fingerprint from April's own pipeline, which I enjoyed more than I should have.)
One trace per step, one span per completion, one child span per reward, 57 spans a step. Every number rides as an attribute: reward.score, grpo.advantage, completion.confidence, hack.signature. Timestamps are real and current, on purpose: SigNoz drops spans older than its retention window, so backdating to April would have been dishonest and silently broken. training.step becomes the dashboard x axis instead. With a 2.5 second sleep between steps, the run took a little under eight minutes, and six charts I last watched in April drew themselves again live.
The collapse
Something I never noticed in three months of owning this data: between steps 18 and 28, nearly everything craters at once, then recovers. Filtering traces to that window returned exactly 11 traces, one per step. Reading 88 completions span by span does not scale, so I exported the quality spans to CSV and joined them to the parent spans with a short script. (Joining a parent attribute with a child score in one query beat me on my setup; the export did fine.)
Step 18: clean, all eight between 0.173 and 0.260. Step 22: eight zeros out of eight.
First theory: overconfidence. Calibration is 1 minus (confidence minus quality) squared, so with quality at zero it lands at exactly zero only when confidence is 1.0. Then the real attribute on the parent span: completion.confidence was 0. A parser fallback, not a confession. Dead.
Second theory: truncation. Step 22's completions all end mid-word, one cut off inside "cited_artifacts": ["KB-0. Except steps 20 and 21 looked like complete, properly closed JSON and still carried five zeros each. I had also just misread one completion while eyeballing, which is when I stopped trusting my eyes.
So I stopped reading and started parsing. json.loads on all 88, fence stripped. 47 failed. Parse failures next to zero scores, step by step:
parse failures: 0 1 5 5 8 5 4 7 4 6 2
zero scores: 0 1 5 5 8 5 4 7 4 6 2
Identical, all eleven steps, from two independent sources: the SigNoz export and the raw parquets. Every zero in that window is a parse failure. Nothing left over.
83% of failures were "Invalid control character". The exact bytes at one failure position:
' resolve this issue, follow these steps:\n\n1. Use `'
Two literal newline bytes inside a JSON string. A paragraph break, right before a numbered list.
That is the whole bug. GRPO pushed the model toward longer, better organized answers, and the model obliged: section breaks, numbered steps. Raw newlines inside a JSON string are invalid JSON, and my pipeline zeroes four rewards at once when it cannot parse. The model was penalized, hard, for becoming a better technical writer. Step 18 answers are short single paragraphs and parse clean. Step 22 answers look like documentation and score zero. No curve shows this. You have to read the completion next to its scores. (Steps 27 and 28 add a second mode, genuine truncation. Two bugs, not one.)
Two more questions the attributes made possible. Confidence above 0.8 with quality below 0.2: 143 completions, 94% stating exactly 0.85. Not overconfidence, a habit: the model says 0.85 the way people say "pretty sure". And grpo.advantage above 1 with quality below 0.15: 23 completions the optimizer rewarded for being the least bad of their group of eight. No WandB view answers either question.
The alert I needed in April
I also resurrected the April bug itself. The v1 parsimony function (rewards anything under 200 or over 400 characters, zeroes the sane middle) now runs beside the fixed version, and hack.signature goes true when v1 scores 0.9 or higher while real quality sits under 0.10. A trace-based alert on that count, wired to Slack, fired: seven flagged completions across five traces, one step carrying three. It resolved itself ten minutes later once the pass ended. This is a reconstruction against real completions, not the original hacked rollouts, and v1 trips on length alone: a tripwire, not proof of an incident. Still. In April this discovery cost me until 4:30 in the morning. Now it is a Slack notification.
(The "min" after the count in the message is a SigNoz unit label quirk; it is a plain count.)
Three independent queries (grounding zeros after step 50, the confidence habit, the optimizer gap) kept converging on the same two tickets, TRAIN-00044 and TRAIN-00046. Whatever those two are supposed to teach, my model never learned it, and no curve ever told me.
Last check: I pointed Claude Code at SigNoz's MCP server from a fresh session with no access to my notes and asked, in plain English, for the overconfident low quality tickets. It rediscovered the schema, worked out the parent child join on its own, and returned in about 18 minutes: 443 matches across 35 tickets (all runs I ever sent, smoke tests included, so wider than my 143). The same two tickets again.
The feature, named
The brief asks for the feature I found most useful. Not the trace waterfall. The Query Builder, specifically what it does with attributes it has never seen: reward.score, grpo.advantage, hack.signature are words I made up, and the moment they arrived they were filterable, chartable, and alertable like any built-in field. Every finding in this post is a query over my own vocabulary. The tool that finds your bugs is the one that speaks the terms your system already thinks in.
What I take from this
I fine tune models, which means I live inside aggregate curves. Loss down, mean reward up, ship it. When something breaks mid-run you squint at a chart, or you bolt a list into a reward function and read raw text at 4 AM. That is the state of the art for a lot of RL fine tuning, and it is half blind. The first time I clicked into one completion and saw the exact text, the ticket, the step, and all six scores in one panel, I sat back in my chair. I had never been able to ask that question before. The build was one continuous seven hour overnight sitting. I kept going because every query got me one step closer, and honestly it was fun, which I did not expect from an observability tool.
If you fine tune with reward functions: your rewards and your model's behavior are telemetry. Treat them that way. And keep your wrong theories in the writeup. Mine both died against a specific number, and the final claim is only trustworthy because they did.
What's next
Same pipeline, pointed at live inference instead of stored completions: that is my build for the main hackathon starting July 20.
Repo with pipeline, casting.yaml, and full investigation logs: REPO LINK.
Environment, trained model, and the original 4 AM story: HF SPACE - MODEL - APRIL BLOG.
Note on AI usage
The direction, architecture, SigNoz exploration, screenshots, and the discovery itself are mine. Instrumentation scaffolding and analysis scripts were written with AI (Claude Code) assistance under my direction and reviewed line by line. Every finding was verified against real data, including two of my own hypotheses that turned out wrong and stayed in this post.
Top comments (2)
The fact that your model was being punished for writing better is a fascinating example of how reward functions can have unintended consequences, and I appreciate the transparency in sharing this experience. The use of SigNoz as an observability platform to re-evaluate the training run and identify the issue is a great example of how important it is to have robust monitoring and evaluation pipelines in place. I'm curious to know more about how you plan to modify your reward functions to avoid similar issues in the future, particularly with regards to the parsimony reward having an inverted branch that led to the model writing about film history instead of fixing servers.
Thanks for reading. One clarification, since I compressed two failures into one post: the inverted parsimony branch was the April bug (the Star Wars one), and it was fixed before this run by rewriting the function as a word count band with unit tests against golden completions. A reward function is code; it deserves tests like code. The collapse in this post is a different failure: a genuine improvement in writing style breaking a strict JSON contract.
For that one, three changes. First, stop hard zeroing four rewards on a parse failure. json.loads with strict=False accepts control characters inside strings, so the model could have been graded on content while a separate format reward taxed the formatting mistake. A cliff gives the optimizer nothing to climb; a slope does. Second, split the contract: prose in a fenced text block, machine fields in a small JSON object, so becoming a better writer can never collide with staying parseable. Third, the boring one that matters most: run the instrumented grading pipeline during training, not three months after. The alert took ten minutes to build. It would have caught both bugs the night they happened.