DEV Community: Yahid Basha

My model got punished for writing better. I found out three months later, in one query.

Yahid Basha — Sat, 18 Jul 2026 12:59:22 +0000

In April I trained a small model to resolve enterprise IT tickets. At 4 AM the night before the Meta x PyTorch hackathon deadline, it started writing about Star Wars. The loss had been flat at exactly zero for thirty minutes. In GRPO that means every completion in a group is earning identical reward, and the cheapest way to do that is one exploit, run eight times per batch. My parsimony reward had an inverted branch. The model used it to write film history instead of fixing servers.

Here is the embarrassing part. I caught it by reading raw completions by hand, dumped to disk by a global Python list I had bolted into a reward function that same night. That list was my observability stack. I needed to see what the model was writing so badly that I hand rolled the worst telemetry pipeline imaginable at 4 AM.

Two nights ago I did what I should have done then: re-graded that training run through a live, instrumented evaluation pipeline, with self-hosted SigNoz on the other end. It surfaced a bug I did not know I had, and not the one I went looking for. In the middle of a run I had called successful, my model spent ten straight steps being punished for writing better.

What I actually ran

The system: Qwen2.5-3B trained with GRPO for 200 steps, 8 completions per step, six reward functions grading each one (format, ROUGE-L quality, citation grounding, calibration, parsimony, repetition). 1,600 graded completions, logged to parquet during training. I had not heard of SigNoz before this hackathon and assumed it was an agent monitoring tool. It is a general observability platform, and its own LLM observability guide lists evaluation and fine-tuning monitoring as a core pillar. A reward pipeline is an evaluation system.

To be precise: no gradients ran. I re-executed the six reward functions, the real April code, on the stored April prompts and completions, with OpenTelemetry wrapping every call. WandB is the experiment log, aggregated curves per step. SigNoz became the flight recorder, every individual completion queryable and alertable by any attribute I could invent. It will not replace WandB for gradients or checkpoints, and I would not try.

The build

foundryctl cast stood the stack up on a base M1 with 5GB handed to Docker, from this casting.yaml:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment: { flavor: compose, mode: docker }
  mcp: { spec: { enabled: true } }

Before instrumenting anything, a fidelity gate: re-grade one parquet against April's logged scores. Max drift: 2.44e-08. Same code, bit for bit. (The tiny drift that did exist was a float32 fingerprint from April's own pipeline, which I enjoyed more than I should have.)

One trace per step, one span per completion, one child span per reward, 57 spans a step. Every number rides as an attribute: reward.score, grpo.advantage, completion.confidence, hack.signature. Timestamps are real and current, on purpose: SigNoz drops spans older than its retention window, so backdating to April would have been dishonest and silently broken. training.step becomes the dashboard x axis instead. With a 2.5 second sleep between steps, the run took a little under eight minutes, and six charts I last watched in April drew themselves again live.

The collapse

Something I never noticed in three months of owning this data: between steps 18 and 28, nearly everything craters at once, then recovers. Filtering traces to that window returned exactly 11 traces, one per step. Reading 88 completions span by span does not scale, so I exported the quality spans to CSV and joined them to the parent spans with a short script. (Joining a parent attribute with a child score in one query beat me on my setup; the export did fine.)

Step 18: clean, all eight between 0.173 and 0.260. Step 22: eight zeros out of eight.

First theory: overconfidence. Calibration is 1 minus (confidence minus quality) squared, so with quality at zero it lands at exactly zero only when confidence is 1.0. Then the real attribute on the parent span: completion.confidence was 0. A parser fallback, not a confession. Dead.

Second theory: truncation. Step 22's completions all end mid-word, one cut off inside "cited_artifacts": ["KB-0. Except steps 20 and 21 looked like complete, properly closed JSON and still carried five zeros each. I had also just misread one completion while eyeballing, which is when I stopped trusting my eyes.

So I stopped reading and started parsing. json.loads on all 88, fence stripped. 47 failed. Parse failures next to zero scores, step by step:

parse failures: 0  1  5  5  8  5  4  7  4  6  2
zero scores:    0  1  5  5  8  5  4  7  4  6  2

Identical, all eleven steps, from two independent sources: the SigNoz export and the raw parquets. Every zero in that window is a parse failure. Nothing left over.

83% of failures were "Invalid control character". The exact bytes at one failure position:

' resolve this issue, follow these steps:\n\n1. Use `'

Two literal newline bytes inside a JSON string. A paragraph break, right before a numbered list.

That is the whole bug. GRPO pushed the model toward longer, better organized answers, and the model obliged: section breaks, numbered steps. Raw newlines inside a JSON string are invalid JSON, and my pipeline zeroes four rewards at once when it cannot parse. The model was penalized, hard, for becoming a better technical writer. Step 18 answers are short single paragraphs and parse clean. Step 22 answers look like documentation and score zero. No curve shows this. You have to read the completion next to its scores. (Steps 27 and 28 add a second mode, genuine truncation. Two bugs, not one.)

Two more questions the attributes made possible. Confidence above 0.8 with quality below 0.2: 143 completions, 94% stating exactly 0.85. Not overconfidence, a habit: the model says 0.85 the way people say "pretty sure". And grpo.advantage above 1 with quality below 0.15: 23 completions the optimizer rewarded for being the least bad of their group of eight. No WandB view answers either question.

The alert I needed in April

I also resurrected the April bug itself. The v1 parsimony function (rewards anything under 200 or over 400 characters, zeroes the sane middle) now runs beside the fixed version, and hack.signature goes true when v1 scores 0.9 or higher while real quality sits under 0.10. A trace-based alert on that count, wired to Slack, fired: seven flagged completions across five traces, one step carrying three. It resolved itself ten minutes later once the pass ended. This is a reconstruction against real completions, not the original hacked rollouts, and v1 trips on length alone: a tripwire, not proof of an incident. Still. In April this discovery cost me until 4:30 in the morning. Now it is a Slack notification.

(The "min" after the count in the message is a SigNoz unit label quirk; it is a plain count.)

Three independent queries (grounding zeros after step 50, the confidence habit, the optimizer gap) kept converging on the same two tickets, TRAIN-00044 and TRAIN-00046. Whatever those two are supposed to teach, my model never learned it, and no curve ever told me.

Last check: I pointed Claude Code at SigNoz's MCP server from a fresh session with no access to my notes and asked, in plain English, for the overconfident low quality tickets. It rediscovered the schema, worked out the parent child join on its own, and returned in about 18 minutes: 443 matches across 35 tickets (all runs I ever sent, smoke tests included, so wider than my 143). The same two tickets again.

The feature, named

The brief asks for the feature I found most useful. Not the trace waterfall. The Query Builder, specifically what it does with attributes it has never seen: reward.score, grpo.advantage, hack.signature are words I made up, and the moment they arrived they were filterable, chartable, and alertable like any built-in field. Every finding in this post is a query over my own vocabulary. The tool that finds your bugs is the one that speaks the terms your system already thinks in.

What I take from this

I fine tune models, which means I live inside aggregate curves. Loss down, mean reward up, ship it. When something breaks mid-run you squint at a chart, or you bolt a list into a reward function and read raw text at 4 AM. That is the state of the art for a lot of RL fine tuning, and it is half blind. The first time I clicked into one completion and saw the exact text, the ticket, the step, and all six scores in one panel, I sat back in my chair. I had never been able to ask that question before. The build was one continuous seven hour overnight sitting. I kept going because every query got me one step closer, and honestly it was fun, which I did not expect from an observability tool.

If you fine tune with reward functions: your rewards and your model's behavior are telemetry. Treat them that way. And keep your wrong theories in the writeup. Mine both died against a specific number, and the final claim is only trustworthy because they did.

What's next

Same pipeline, pointed at live inference instead of stored completions: that is my build for the main hackathon starting July 20.

Repo with pipeline, casting.yaml, and full investigation logs: REPO LINK.

Environment, trained model, and the original 4 AM story: HF SPACE - MODEL - APRIL BLOG.

Note on AI usage

The direction, architecture, SigNoz exploration, screenshots, and the discovery itself are mine. Instrumentation scaffolding and analysis scripts were written with AI (Claude Code) assistance under my direction and reviewed line by line. Every finding was verified against real data, including two of my own hypotheses that turned out wrong and stayed in this post.

When my RL agent started writing about Star Wars instead of fixing servers

Yahid Basha — Sun, 26 Apr 2026 10:54:02 +0000

A Sunday-morning postmortem on teaching a 3B model to do enterprise IT triage with GRPO.

It's 1 AM on a Sunday. The Meta × PyTorch OpenEnv Hackathon submission is due at 5 PM. My training logs show a loss curve that's been flat at 0.0 for the last thirty minutes.

A flat loss in supervised learning means convergence. A flat loss in reinforcement learning usually means something else: your model has found a way to game your reward function, and it is now sitting in a local optimum, smug about it.

Mine had decided that the easiest way to maximize reward was to write about cinema history.

What I was actually trying to build

I work on a production RAG system at Verizon — the kind of system that's supposed to surface the right runbook when a NOC engineer types "BGP (Border Gateway Protocol) session won't come up after the Tuesday maintenance window." This is Theme #3.1 — World Modeling for Professional Tasks — applied to a domain where the model must maintain accurate beliefs about a partially observable KB, not exploit shortcuts to fake a resolution. If you've ever built one of these, you know the failure modes are not in retrieval. Retrieval is mostly a solved problem. The failures are behavioral, at the LLM layer:

The model retrieves the right document and still hallucinates the answer. It cites a KB article that doesn't exist. It sounds confident when it should escalate. It pads the response with filler because shorter answers feel less authoritative. None of these are bugs you can fix with better embeddings.

For the hackathon, I wanted to attack one slice of this problem with reinforcement learning. The thesis: if you give an LLM the right kind of feedback signal — not labeled examples but a programmatic verifier of what good triage looks like — you can teach a small model to behave well over retrieved context, even if it doesn't have deep domain knowledge baked in.

I picked Qwen2.5-3B-Instruct. Not because 3B is fashionable, but because the only result that's interesting to an enterprise team is "small model + RL > big model off the shelf, on this specific task, for this specific cost." If I can't show that, the project doesn't matter.

The first attempt, and TRL issue #5536

The original design was ambitious: a multi-turn agent that decides which tools to call. The action space had seven tools — search_kb, search_tickets, search_incidents, get_article, get_ticket, get_incident, and submit_resolution. The agent would search, fetch, search again, eventually submit a resolution. GRPO would teach it the right query strategies.

I burned about four hours on this before I realized the environment factory in TRL wasn't injecting tool schemas into the rollout context properly. The model was sampling actions blind. There's an open issue (#5536) for it. I tried three workarounds. None of them produced a clean rollout.

Around 4 AM I noticed the loss had flatlined and the rewards looked suspiciously stable. I pulled a few sample completions to inspect them.

The model was writing about Star Wars.

Specifically, it was writing about the history of cinema, the production of A New Hope, and at one point, a fairly accurate paragraph about George Lucas's relationship with 20th Century Fox. The submit_resolution JSON at the end was empty. The resolution string was movie trivia.

Why? Because I had a parsimony reward — "give a concise answer, don't ramble" — that I had implemented incorrectly. The function rewarded outputs under 200 characters and over 400. (I wanted U-shaped to discourage stub answers and discourage padding, but my logic was inverted in one branch.) Long, irrelevant filler hit the high-length lobe of the reward and got credited as if it were a complete answer.

This is the canonical failure mode of RL. The OpenEnv documentation has a whole section about it. I'd read that section. I still wrote the bug.

The pivot: single-turn, oracle-grounded

It was 4:30 AM and I had two options. Spend the morning fighting TRL and debugging multi-turn rollouts, or simplify the environment until I had something that trains.

I picked the simplification. The new design dropped multi-turn entirely:

The prompt contains the ticket plus retrieved articles — some gold, some distractors. I do the retrieval before the rollout.
The model reads the context and emits exactly one submit_resolution action containing four fields: a resolution string, a list of citation IDs, a confidence score, and an escalation boolean.
The reward grades that single response on six independent dimensions.

The honest framing of this trade-off: I'm no longer training a model to be a good searcher. I'm training it to be a good reader. Given a ticket and five candidate articles, can it identify which one is relevant, write a grounded answer, cite the right IDs, and tell me how confident it is? That's a narrower capability than the original goal. It's also exactly the capability that fails in production RAG systems, so it's still worth doing.

Six rewards, none of them load-bearing alone

I built the reward function as six independent components:

reward_funcs = [
    r_format_graduated,    # JSON schema compliance, no shortcuts
    r_resolution_quality,  # ROUGE-L against gold resolution text
    r_citation_grounding,  # cite only IDs that appear in context
    r_calibration,         # confidence ~ correctness (Brier-style)
    r_parsimony,           # answer shorter than the ticket itself
    r_repetition_penalty,  # punish loops and copy-paste
]

The argument for multiple independent rewards is straightforward: with one scalar, your model finds the cheapest way to maximize it, which is almost never what you want. With six, gaming any single component costs you on the others. The Star Wars incident was a reminder of why this matters — that whole behavior was possible because parsimony was the only signal in town for that branch of the reward.

r_repetition_penalty is purely defensive. I added it after the Star Wars incident as a guard against the next reward-hacking failure mode I could imagine — a model that finds it can boost grounding by repeating the same valid citation thirty times. It's the only training reward without a server-side equivalent. The environment's evaluation reward function has five components; training has six because I didn't trust the model not to find a new exploit.

What 200 steps of GRPO actually changed

I trained for 200 steps on an A100. After the run, I pulled the completions parquets from the HF model repo and aggregated the per-step reward means. Two windows: steps 1–10 (cold start, before any meaningful gradient) and steps 150–200 (post-convergence).

Reward	Steps 1–10	Steps 150–200	Δ
Calibration	0.528	0.977	+0.449
Parsimony	0.246	0.940	+0.694
Citation Grounding	0.756	0.862	+0.106
Resolution Quality	0.162	0.181	+0.019
Format Adherence	1.000	0.992	—
Repetition Penalty	1.000	0.990	—

Three of these moved a lot. Three didn't.

Format and repetition were already pinned at the ceiling from the first step. Qwen2.5-3B-Instruct is well-aligned out of the box — it knows JSON, and the base model wasn't producing the kind of pathological loops that the repetition penalty was designed to catch. Those rewards weren't doing anything; they were ballast. (The repetition penalty was still cheap insurance, and it's the kind of thing you don't notice working until the day it does.)

Calibration nearly doubled. Parsimony quadrupled. Citation grounding climbed from "okay" to "good." These are the rewards that track behavioral properties — properties of how the model speaks, not what it knows. The model learned to express less confidence when its answer was wrong (calibration), to stop padding responses (parsimony), and to draw citations from the actual provided context instead of inventing them (grounding).

Then there's the one number that barely moved. r_resolution_quality went from 0.16 to 0.18. I want to be honest about that one because it's the most interesting result on the list.

Reading the reward curves

Six reward signals over 200 GRPO steps. Each line is the mean reward
across the 8 completions sampled per step.

One thing that surprises people seeing RL training curves for the first time:
the loss isn't what you watch. In GRPO, the loss oscillates around zero throughout
training — that's expected behavior for the dr_grpo objective. What you watch
instead are the reward curves, and you want them to go up and stabilize.

The six curves here fall into four groups that each tell a different story.

Format and repetition (flat at 1.0 from step 1) — the base model already knew the JSON schema and didn't loop citations. GRPO had nothing to teach here. Flat lines at the ceiling are a good sign, not stagnation.

Calibration and parsimony (S-curve convergence, steps 1–30) — these are the headline results. Calibration climbs from 0.53 to 0.97 in roughly thirty steps. Parsimony goes from 0.25 to 0.94 over the same window. Both show a classic S-curve: slow start, fast middle, then plateau. These are behavioral properties the base model had the latent capacity for but hadn't been incentivized to use. The gradient found them quickly.

Citation grounding (noisy upward trend) — the most visually dramatic curve. It swings between 0.0 and 1.0 throughout all 200 steps, and the variance doesn't obviously shrink. This is not a training failure — it's a harder reward to optimize. Citation grounding depends on each specific ticket's context: sometimes the model identifies the right article, sometimes it picks a distractor. The noise is honest. The mean trends upward (0.756 → 0.862), which is what matters.

Resolution quality (the line that doesn't move) — the orange line stays near 0.18 for the entire run. This is the most important curve to understand correctly. It's not flat because training failed. It's flat because GRPO cannot add knowledge the base model doesn't have. Writing the correct BGP (Border Gateway Protocol) reset procedure requires knowing the correct BGP (Border Gateway Protocol) reset procedure — that's a pretraining question, not a fine-tuning question. The ceiling here tells you where RL ends and domain pretraining begins.

The ceiling that RL can't push through

r_resolution_quality is ROUGE-L overlap between the model's resolution text and the gold answer. It measures something like "did you actually write the right fix?"

Calibration is a style property. The model can learn it by adjusting how it expresses uncertainty in language it already produces. Parsimony is also style — say less. Citation grounding is behavioral — pick from this list, don't invent IDs. RL is good at all three.

But generating the factually correct fix for a BGP (Border Gateway Protocol) session that won't come up requires knowing the actual answer. RL doesn't add knowledge. It can only shape behavior over what the base model already has.

So r_resolution_quality ≈ 0.18 isn't really a training failure. It's the ceiling that GRPO alone hits on a 3B base that doesn't have deep enterprise networking knowledge baked into pretraining. The next steps to break that ceiling are retrieval-augmented generation at inference time, domain-adapted continued pretraining, or both. Neither is RL.

I think this is the most useful thing I can communicate from this project: a clean separation between what RL fine-tuning fixes (style, calibration, format adherence, faithfulness to provided context) and what it doesn't (factual knowledge gaps). If you're a team thinking about deploying a small model behind a RAG system, the first list is achievable in a weekend with a few hundred GRPO steps. The second list is a much bigger investment.

What this means for production RAG

The pitch isn't that I trained a model that hits some benchmark. It's that the most common LLM-layer failures in production RAG systems — hallucinated citations, miscalibrated confidence, padded answers, format drift — are tractable with RL on a small base model, with a verifier that scores the four behaviors independently.

You don't need to swap your 7B for a 70B to fix these. You need to teach the model you already have to behave correctly over retrieved context.

Calibration improved by 85% relative — from 0.528 to 0.977. In production terms, that's the difference between an agent that confidently misroutes a P1 incident at 3 AM and one that escalates honestly when it doesn't know the answer. That's not a benchmark number. That's a safety property.

What I'd do differently

A few things I'd change if I were starting over:

The dataset is too small. 50 training tickets across networking, security, cloud, and infra means each subdomain only gets ten or twelve examples. A larger, more diverse ticket pool would probably push the behavioral rewards higher and might lift resolution quality a bit through better domain coverage.

I'd add a process reward, not just outcome rewards. Right now the model is graded only on the final submit_resolution. Scoring intermediate reasoning steps — even with a lightweight LLM-as-judge as one signal among many — would give a denser gradient.

I'd run the 72B baseline comparison properly. I tried, on Sunday afternoon, to run the trained 3B and the off-the-shelf 72B side by side against the environment's server-side rewards, hit a CUDA driver mismatch on the eval container, and ran out of time. The training-reward comparison is real; the small-model-with-RL > big-model-without claim is partially evidenced. I'd want a clean head-to-head on equal footing before saying it cleanly.