DEV Community: Vaibhav Mittal

I Fixed a Schema Mismatch. The Negative Result Survived.

Vaibhav Mittal — Sat, 08 Aug 2026 05:43:13 +0000

An external reviewer found a real mismatch in my experiment.

I had tested whether an LLM should generate a numeric string directly or generate a
JSON integer that a deterministic transducer converts back to the caller's string
contract.

The Llama result was negative: the integer representation lost ten net correct
answers on 150 unseen GSM8K items.

But the two model-facing languages were not exactly equivalent.

My string control accepted integers, decimals, fractions, comma grouping, and
leading zeros. The compiler transform supported only canonical signed integers.

That meant I had tested a representation change plus a domain-narrowing change.

I did not explain the mismatch away. I preregistered one correction, generated one
new 150-row control arm, reused the frozen treatment, audited every disagreement,
and accepted the outcome.

The negative result survived.

Exact canonical string control: 92/150 correct. Frozen integer treatment: 82/150.
Paired effect: -6.7 percentage points, with interval [-12.7, -0.7].

This closes the default optimizer thesis for this project. More importantly, it
shows why model-facing schemas must be treated as semantic context, not transparent
serialization wrappers.

Vaibhav701161 / constrained-decoding-lab

Constrained Decoding Under Matched Conditions

A controlled, artifact-validated study of how JSON prompting, grammar-constrained decoding, and output-field order affect mathematical accuracy and schema compliance.

Progress presentation

Watch this presentation for an overview of the project's progress, key findings and results to date.

Watch on YouTube

Central result

Constrained decoding solved the formatting problem, but it did not preserve all of the model's recoverable mathematical accuracy. On Qwen2.5-7B, prompt-only JSON achieved 79.6% recoverable accuracy and 0% schema compliance. Outlines and XGrammar each achieved 61.2% recoverable accuracy and 100% schema compliance. The paired semantic effect was -18.4 percentage points for both backends (exact McNemar p = 0.003906).

This is not a claim that constrained decoding is universally harmful. It is a controlled…

View on GitHub

The mismatch

The original Llama control used this broad numeric-string grammar:

^-?(?:(?:\d+|\d{1,3}(?:,\d{3})+)(?:\.\d+)?|(?:\d+|\d{1,3}(?:,\d{3})+)/(?:\d+|\d{1,3}(?:,\d{3})+))$

It accepts values such as:

"001"
"1.0"
"36/2"
"1,000"
"-5.5"

The safe compiler transform accepts only:

^-?(?:0|[1-9][0-9]*)$

It maps exactly between:

external JSON string "-12"
        |
        v
internal JSON integer -12
        |
        v
deterministic base-10 stringification
        |
        v
external JSON string "-12"

No rounding, parsing heuristic, sign repair, or second model call is allowed.

Eight actual broad-control outputs were outside the canonical language: six on the
fresh set and two on the bridge set. Examples included "13.33", "55.5",
"94.98", and "36/2".

All eight were incorrect in both arms. So the mismatch did not directly explain the
ten-answer Llama deficit. It still blocked the strongest claim about the exact
compiler-supported transform.

That distinction matters. A result can be directionally meaningful and still need a
targeted correction before supporting its most specific interpretation.

The only new run I allowed

I froze the correction before generation:

Component	Frozen value
Model	`meta-llama/Llama-3.2-3B-Instruct`
Revision	`0cb88a4f764b7a12671c53f0838cd831a0843b95`
Dataset	Same 150 previously unseen GSM8K test items
Backend	XGrammar 0.2.3
Decoding	Greedy, seed 0, FP32
Token budget	256 maximum new tokens
Prompt	Byte-identical raw prompt
New arm	Exact canonical signed integer string
Treatment	Existing frozen integer artifact
Exclusions	None

The frozen treatment SHA-256 was:

298d1a38ad8d95d89ca97ab1f98d14bef4853342bf388d080f57f06de9c47342

The new run was not allowed to change the model, prompt, backend, package versions,
precision, decoding settings, holdout, or treatment. It was also not allowed to add
a bridge arm or Outlines arm.

This was a correction, not a search for a favorable result.

Semantic-blind canary and resumability

The first five rows were evaluated only for operational integrity:

same item IDs and order;
byte-identical prompts;
one chat-template application;
frozen model and tokenizer revisions;
matching package and GPU environment;
no duplicate rows;
no exceptions or cap hits;
100% internal and external validity.

The canary did not inspect correctness before expansion.

One operational interruption happened after row 23 when the local Modal client lost
its heartbeat. Every completed row had already been committed to the remote evidence
volume. I relaunched in detached resume mode, starting at row 24. No completed row
was regenerated.

The final new artifact contains 150 rows in the frozen dataset order, with zero
errors, zero cap hits, zero internal-schema failures, and zero external-schema
failures.

Result

Outcome	Canonical string control	Frozen integer treatment
Contract-valid correct	92/150, 61.3%	82/150, 54.7%
Semantic correct	92/150, 61.3%	82/150, 54.7%
Final external valid	150/150, 100.0%	149/150, 99.3%
Internal schema valid	150/150, 100.0%	149/150, 99.3%
Errors	0	0
Token-cap hits	0	1
Mean generated tokens	78.2	78.9

The paired transition matrix was:

	Treatment correct	Treatment wrong
Control correct	76	16
Control wrong	6	52

So the treatment had six unique wins and sixteen unique losses.

The treatment-minus-control estimate was -6.7 percentage points. The exact paired
bootstrap 95% interval was [-12.7, -0.7] points. The exact two-sided McNemar test
was p = 0.05248.

Those two uncertainty summaries answer related but different questions. I report
both rather than selecting the one with the cleaner threshold. The deterministic
paired bootstrap interval excludes zero. The exact McNemar result narrowly misses
0.05.

The preregistered decision did not depend on calling the result statistically
significant. It said that if the exact canonical control still beat the frozen
treatment, the optimizer thesis would close. The control beat it by ten net items.

The treatment's one cap hit remained in the denominator. That item was wrong in both
arms, so it did not create a discordant pair.

What changed relative to the broad control

The canonical grammar did change model behavior:

134/150 raw outputs were byte-identical to the broad control;
140/150 normalized final answers were identical;
16 raw outputs changed;
one item changed from wrong to correct;
one item changed from correct to wrong;
aggregate control correctness remained 92/150.

The six noncanonical fresh-set values disappeared, as required. But canonicalization
did not improve aggregate accuracy or remove the treatment deficit.

This is an important negative mechanism result. The broad language mismatch was
real, but it was not the cause of the overall direction.

I inspected every discordant item

There were 22 discordant pairs after the correction. I inspected all of them against
the question, reference answer, raw control output, and raw treatment output.

Category	Count
Problem-interpretation change	10
Reasoning and final-answer inconsistency	8
Arithmetic regression	3
Arithmetic correction	1
Sign or lexical-boundary change	0
Parser or validator issue	0
Truncation	0

The dominant pattern was not an inability to emit a minus sign.

For example, in one control-only win both conditions produced the same correct
reasoning that Sam should receive 20 feet of fence. The integer treatment emitted
40 anyway.

In another, both conditions derived two trays of eggnog. The treatment selected the
intermediate count of ten glasses as its final answer.

In a treatment-only win, the string control stopped after calculating 4,750
remaining graduation seats. The integer treatment completed the requested division
by 950 graduates and returned five tickets each.

These are trajectory-level changes. A one-field schema rewrite can alter which
steps are retained, which arithmetic is executed, and which number becomes final.

Contract preservation after generation does not imply behavioral equivalence
during generation.

Engineering corrections beyond the run

The review also exposed engineering work worth fixing regardless of the result.

I made the following changes:

Defined the canonical signed-integer regex once and generated the experimental schema through the same compiler rewrite path.
Added one-command replay that reparses raw outputs, revalidates schemas, retransduces objects, recomputes task and execution scores, and reconstructs paired summaries.
Made integer-to-string rewrites fail closed when enum, const, bounds, or multipleOf would otherwise be dropped.
Replaced generic float coercion with exact integer and deliberate decimal comparison.
Added pyproject.toml, MIT licensing, dependency extras, and CI on Python 3.11 and 3.12.
Split lightweight replay dependencies from GPU and backend dependencies.

In a clean lightweight environment, 129 tests passed and eight generation or backend
tests skipped explicitly. The replay command reconstructed 464 earlier GSM8K and
tool-dispatch rows with zero row-score mismatches and zero paired-summary mismatches.

The full pinned environment currently reports 143 passed and three Llama prompt
parity tests skipped locally because they require the frozen Modal image.

The product decision

I started by asking whether caller contracts could be compiled into model-friendlier
internal representations that improve quality.

The corrected Qwen experiment produced a positive estimate:

18/49 canonical contract-valid correct
        to
24/49 integer + transducer contract-valid correct

But the exact canonical Llama correction went the other way:

92/150 canonical contract-valid correct
        to
82/150 integer + transducer contract-valid correct

A separate 30-item BFCL-derived deterministic tool-dispatch and post-state pilot also
found no evidence of practical benefit. That pilot did not execute arbitrary
business functions, and its interval was wide, so I do not claim general tool-calling
harm.

The project now has this primary loop:

External contract
        |
        v
Candidate model-facing representations
        |
        v
Frozen paired workload
        |
        v
Matched constrained generations
        |
        v
External validation and deterministic dispatch
        |
        v
Paired correctness, uncertainty, and complete audit
        |
        v
Workload-scoped recommendation or refusal

The product is a contract-sensitivity evaluation harness.

The linter is secondary. It can say:

This contract contains a representation-sensitive boundary.
Measure it before deployment.

It cannot say:

Convert this string to an integer; accuracy will improve.

Reproduce the evidence without a GPU

The default replay path is intentionally lightweight:

git clone https://github.com/Vaibhav701161/constrained-decoding-lab.git
cd constrained-decoding-lab
git checkout exp/canonical-schema-correction

uv venv .venv --python 3.12
uv pip install --python .venv/bin/python -e ".[dev]"
source .venv/bin/activate

python -m pytest
python scripts/replay_artifacts.py \
  --scope all \
  --out /tmp/replay-validation.json

The complete correction record includes:

Final lesson

The compiler utility is still useful. It preserves the supported external contract,
fails closed, and makes representation alternatives executable.

What failed was the assumption that a model-friendlier representation would be a
portable quality optimization.

That is the result I am keeping.

If a schema rewrite changes the tokens available to a model, it can change the whole
generation trajectory. Measure it on the actual model and workload. Preserve every
regression. Refuse to search until the story becomes positive.

That is a stronger engineering system than an optimizer built from one encouraging
experiment.

The Optimization Worked on Qwen. It Failed on Llama and Tool Calls.

Vaibhav Mittal — Fri, 07 Aug 2026 19:56:08 +0000

Correction update, 8 Aug 2026

An external review found that the Llama string control described below accepted
decimals, fractions, comma grouping, and leading zeros, while the safe compiler
supported only canonical signed integers. I preregistered one exact canonical
control arm and reused the frozen treatment. The result remained negative: 92/150
versus 82/150, a -6.7 point paired effect with interval [-12.7, -0.7], 6 wins, and
16 losses. This resolves the mismatch and closes the default optimizer thesis. See
the complete correction decision.

I had a promising compiler result.

On Qwen2.5-7B, changing one model-facing JSON field from a signed numeric string to a
native integer, then deterministically converting it back to the caller's unchanged
string contract, improved contract-valid GSM8K correctness from 18/49 to 24/49.

The paired estimate was +12.2 percentage points. The transform preserved 100%
external validity. The implementation had property tests, exact inverse
transduction, fail-closed validation, frozen artifacts, and two grammar backends that
produced byte-identical outputs.

It looked like the beginning of a contract compiler for LLMs.

Then I ran the test that mattered:

Does the improvement survive on a different model family, unseen items, and an
executable tool-call task?

It did not.

On 150 unseen GSM8K items with Llama 3.2 3B, the same treatment reduced correctness
from 92/150 to 82/150. On a separate 30-case executable primary sample built from
pinned BFCL V4 cases, it reduced exact executable-call success from 26/30 to 24/30.

The honest conclusion is not that the implementation failed. The transducer and
validator worked perfectly in the practical pilot.

The conclusion is more important:

A contract-preserving representation change is still a semantic intervention on
the model, and a local win is not a portable optimizer.

The code, protocols, raw generations, exact hashes, paired statistics, validators,
and complete discordance audits are public.

Vaibhav701161 / constrained-decoding-lab

Constrained Decoding Under Matched Conditions

A controlled, artifact-validated study of how JSON prompting, grammar-constrained decoding, and output-field order affect mathematical accuracy and schema compliance.

Progress presentation

Watch this presentation for an overview of the project's progress, key findings and results to date.

Watch on YouTube

Central result

This is not a claim that constrained decoding is universally harmful. It is a controlled…

View on GitHub

What was being optimized?

Suppose a caller requires this object:

{
  "name": "adjust_inventory",
  "arguments": {
    "sku": "ABC-14",
    "quantity_delta": "-12"
  }
}

The externally required quantity_delta is a canonical signed integer string. Its
language is:

^-?(0|[1-9][0-9]*)$

The proposed internal contract lets the model emit a native JSON integer:

{
  "name": "adjust_inventory",
  "arguments": {
    "sku": "ABC-14",
    "quantity_delta": -12
  }
}

The runtime then performs an exact inverse:

external JSON Schema
        |
        v
contract IR and applicability analysis
        |
        v
internal JSON Schema
        |
        v
one constrained model generation
        |
        v
deterministic integer-to-string transduction
        |
        v
validation against the original external schema
        |
        v
caller-facing object

There is no second model call. There is no sign repair, rounding, regex extraction,
default insertion, key guessing, or fallback coercion. A generated boolean is
rejected even though Python treats booleans as integer subclasses. Unsupported
schemas fail closed.

This proves a useful engineering property: the internal object can be converted back
without changing the generated sign or magnitude.

It does not prove a model-quality property: that asking the model for an integer
will make it choose better values.

That second claim needed replication.

The evidence before replication

The project started with a matched Qwen2.5-7B study of constrained JSON generation.
Prompt-only JSON achieved 79.6% recoverable GSM8K accuracy but 0% compliance with a
numeric-string answer schema. Outlines and XGrammar achieved 100% schema compliance
but only 61.2% recoverable accuracy.

That -18.4 point semantic difference showed that schema compliance and task
correctness were separate outcomes.

The native-integer transform was a narrow attempt to recover some of the lost
correctness without weakening the caller's contract. After finding and correcting a
double-chat-template risk in the historical runner, I reran both representations
from a frozen source.

The corrected Qwen result was:

Representation	Contract-valid correct	External valid
Signed numeric string	18/49 (36.7%)	49/49
Native integer plus transducer	24/49 (49.0%)	49/49

Paired statistics:

difference:                 +12.2 percentage points
exact bootstrap interval:   [0.0, 26.5] points
treatment-only wins:         9
control-only wins:           3
exact McNemar p:             0.145996

This was valid, positive, and uncertain. The interval touched zero. It was enough to
authorize one independent replication, not enough to say the compiler worked in
general.

I froze the Llama test before generation

The confirmatory question was:

On a non-Qwen model and unseen GSM8K items, does replacing a model-facing signed
numeric string with a native JSON integer improve final external-contract-valid
correctness after deterministic stringification?

The fixed design used:

Component	Frozen value
Model	`meta-llama/Llama-3.2-3B-Instruct`
Exact revision	`0cb88a4f764b7a12671c53f0838cd831a0843b95`
Primary backend	XGrammar 0.2.3
Decoding	Greedy, seed 0, FP32, 256 maximum new tokens
Confirmatory set	150 randomly selected, previously unseen GSM8K test items
Bridge set	Existing cleaned 49-item set
Primary outcome	Paired contract-valid correctness on the fresh 150
Error policy	Every error, cap, invalid object, and transduction failure stays in the denominator
Post-launch exclusions	None

The unseen-set preparation scanned existing JSONL artifacts under results/,
experiments/, and deployment/, removed every previously used source ID, shuffled
the remaining GSM8K test items with seed 20260815, and selected 150. The manifest
records the excluded set, selected IDs, source, split, seed, and hashes.

Control and treatment used one shared generation runtime. Model loading, tokenizer
handling, chat templating, grammar compilation, output decoding, token counting,
latency measurement, error handling, manifests, and resumable writing were
identical. Only the model-facing schema, symbolic representation in the prompt, and
treatment inverse transduction differed.

The first five paired items were an operational canary. Expansion depended on IDs,
ordering, prompt-template depth, outputs, errors, caps, validity, revisions,
environment, signatures, and dataset hashes.

Semantic wins were explicitly forbidden from influencing expansion.

The cross-family result was negative

The complete XGrammar matrix contained 398 generations:

fresh set:   150 control + 150 treatment
bridge set:   49 control +  49 treatment
total:       398 generations

The fresh confirmatory result was:

Metric	String control	Integer treatment
Contract-valid correct	92/150 (61.3%)	82/150 (54.7%)
Final external valid	150/150	149/150
Internal-schema valid	150/150	149/150
Generation errors	0	0
Token-cap hits	0	1

Paired effect:

difference:                 -6.7 percentage points
exact bootstrap interval:   [-12.7, -1.3] points
treatment-only wins:         5
control-only wins:          15
exact McNemar p:             0.041389

The single treatment cap hit stayed in the denominator. It was not retried or
repaired.

The bridge set also moved slightly negative:

control:                    21/49
treatment:                  20/49
difference:                 -2.0 points
interval:                   [-10.2, 6.1] points
wins : losses:               2 : 3

The primary interval was entirely below zero. This was not a merely inconclusive
replication. Under the frozen gate it was Red: no cross-family replication.

The failure audit changed the mechanism story

I manually inspected every discordant item, not a sample of them.

Across the fresh and bridge sets there were 25 discordances:

Category	Count
Problem-interpretation change	12
Reasoning-to-final inconsistency	9
Arithmetic regression	3
Arithmetic correction	1
Sign or lexical-boundary change	0
Parser or validator issue	0
Truncation among discordants	0

The representation change did not simply remove friction at a local quote or minus
boundary. It changed broader generation behavior. Some final answers stopped
matching the model's own reasoning. Other outputs interpreted the same word problem
differently.

That matters because a compiler analogy can become misleading. A conventional
compiler transform is expected to preserve program semantics. Here, changing the
schema shown to the model changes the probability distribution that produces the
semantics in the first place.

The inverse transducer can be exact while the generation intervention is not
semantics-preserving at the model level.

I checked that XGrammar was not the culprit

After freezing the XGrammar result and discordant set, I ran Outlines only on:

all 20 fresh discordant items; and
20 concordant items selected with a frozen seed.

Both representations used the same 40 IDs.

Results:

signed-string outputs matching byte for byte: 40/40
integer outputs matching byte for byte:       40/40
all prompt and scoring fields matching:       80/80
same treatment cap hit reproduced:            yes

This is implementation-parity evidence, not a second statistical replication. It
shows that the negative result on the audited subset was not specific to XGrammar's
implementation.

Red authorized one bounded practical pilot

A negative math replication did not automatically prove the idea useless for tool
calls. The protocol allowed exactly one bounded practical pilot.

The question became:

Does contract alignment improve the probability that an LLM emits an externally
valid call that actually executes with the correct arguments and state?

I used pinned BFCL V4 simple_python cases and official acceptable arguments as the
reference foundation. I did not claim an official BFCL leaderboard evaluation. The
external numeric-string contract was a project-defined adaptation, and deterministic
local wrappers replaced real business functions.

The wrappers had no external side effects. They checked:

exact function selection;
whole-object JSON validity;
internal-schema validity;
deterministic inverse transduction;
reconstructed external-schema validity;
exact typed argument semantics;
dispatch and execution acceptance;
correct post-execution state; and
zero heuristic repairs.

The executable dataset was selected mechanically

An eligible pinned BFCL case had to contain one turn, one function, an object-like
parameter schema, at least one required integer, a single unambiguous ground-truth
call, and only supported schema constructs.

The preparation code found 194 eligible cases out of 400.

The primary pilot selected 30 uniformly without replacement using seed 20260817.
Because negative required integers were rare, all 3 eligible negative cases formed a
separate sign-stress set. They were never pooled into the primary effect.

The two representations produced 66 total generations:

30 primary + 3 stress = 33 cases
33 control + 33 treatment = 66 generations

The three-case canary checked 20 operational invariants without looking at semantic
outcomes. The same files then resumed to the complete set.

The practical pilot was also Red

Every one of the 66 calls was valid JSON, internally schema-valid, externally
schema-valid after treatment transduction, and accepted by the deterministic
dispatcher. There were zero generation errors, cap hits, transduction failures,
execution failures, or heuristic repairs.

That is excellent contract-boundary behavior.

It still did not improve exact calls.

Primary metric	String control	Integer treatment
Executable-contract success	26/30 (86.7%)	24/30 (80.0%)
Internal-schema validity	30/30	30/30
External-schema validity	30/30	30/30
Exact argument semantics	26/30	24/30
Execution acceptance	30/30	30/30
Correct post-execution state	26/30	24/30

Paired statistics:

difference:                 -6.7 percentage points
exact bootstrap interval:   [-20.0, 6.7] points
treatment-only wins:         1
control-only wins:           3
exact McNemar p:             0.625

The sample is small and the interval crosses zero. I am not claiming statistically
proven harm in tool calling generally.

But the preregistered decision rule was directional. A zero or negative estimate, or
losses greater than or equal to wins, was sufficient for Red. Both occurred.

Every practical discordance was inspected

There were five discordant calls across the primary and stress sets.

Treatment correction: emissions

Both conditions emitted the correct integer duration, 3. The control used the
wrong non-integer energy type, solar; the treatment used the accepted value,
renewable.

This was a real correction, but not an integer-boundary repair.

Treatment regression: magnetic permeability

Both required integer fields remained correct. The treatment changed an optional
floating-point permeability from the pinned accepted value to a more precise physical
constant. It was plausible, structurally valid, and wrong under the frozen call
semantics.

Treatment regression: cooking conversion

The user asked to convert 2 pounds to ounces. The control passed quantity=2. The
treatment passed quantity=16, apparently substituting the conversion result for the
function input.

Treatment regression: restaurant threshold

The request required a minimum rating of more than 4, represented by the pinned
argument 4. The treatment emitted 5, strengthening and therefore changing the
requested threshold.

Stress-set correction: quadratic roots

Both conditions preserved a=3, b=-11, and c=-4 exactly. The treatment won only
because a separate string field changed from root_type="real" to
root_type="all".

So even the sole sign-stress repair was not a sign repair.

The complete audit contained two semantic corrections and three semantic
regressions. No discordance came from validation, transduction, dispatch, truncation,
or parsing.

Why I am stopping the optimizer thesis

The three decision gates now tell a coherent story:

Qwen corrected GSM8K:       +12.2 points, positive and uncertain
Llama unseen GSM8K:          -6.7 points, interval below zero
Llama executable pilot:      -6.7 points, small and uncertain

It would be easy to rescue the narrative by pooling the positive three-case stress
set, changing prompts, trying more model families, or emphasizing perfect validity
while hiding incorrect arguments.

That would be research theater.

The external contract was preserved. The model's call meaning was not reliably
improved. The broad claim failed twice after the initial Qwen result.

The optimizing-compiler thesis is closed under the current evidence.

What remains genuinely useful

The negative result does not make the infrastructure useless. It changes what the
product should be.

The repository already has:

canonical contract IR and stable hashing;
deterministic, serializable alignment plans;
conservative transducers with exact inverses;
final validation against the original schema;
typed fail-closed refusals;
unified matched runners;
resumable artifact writing;
source, dataset, environment, and run manifests;
paired bootstrap intervals and exact McNemar tests;
deterministic execution receipts; and
complete item-level discordance audits.

Those components are the foundation for a different product:

1. Schema-risk linter

Identify contract features that may create fragile lexical or generation boundaries.
Do not automatically rewrite them. Explain the risk and the evidence scope.

2. Contract-sensitivity analyzer

Run frozen paired interventions on the user's actual workload. Report validity,
semantic wins, regressions, uncertainty, cap hits, latency, and failure mechanisms
separately.

3. Reproducible measurement harness

Preserve prompts, chat templates, model revisions, schemas, environments, raw
outputs, execution state, and exact artifact hashes so that a structured-output
change can be audited before deployment.

The integer-to-string transducer remains useful as a deterministic utility. It is
not a default quality optimization.

What I learned

Contract correctness and task correctness are orthogonal

The executable treatment reconstructed 100% valid external calls and executed 100%
of them. It still produced fewer exactly correct calls.

Schema changes can perturb untouched fields

Three discordances changed fields whose types were identical across conditions. A
local schema edit can alter the entire autoregressive path.

Backend agreement is not independent replication

Byte-identical XGrammar and Outlines outputs show implementation parity. They do not
turn one model result into two replications.

An uncertainty interval does not replace a frozen decision rule

The small practical pilot cannot prove general harm. It can still fail a
preregistered continuation gate.

Negative replication is product information

The most valuable result was not another positive chart. It was learning which claim
the system could no longer honestly make.

Reproducibility status

Llama primary XGrammar rows:       398/398 validated
Llama discordances audited:         25/25
Outlines parity outputs:            80/80 byte-identical
Executable pilot rows:              66/66 validated
Executable discordances audited:     5/5
Post-launch exclusions:               0
Semantic retries:                     0
Modal billed cost:                 $0.00

The canonical reports are:

I started this stage trying to validate an optimizing compiler.

I finished it with a more defensible system: one that can tell you when a contract
change is safe, when it is harmful, and when the evidence is too weak to decide.

That is less dramatic than claiming a universal optimization.

It is also much more useful.

I Found a Runner Bug, Re-ran 200 Generations, and the Effect Survived

Vaibhav Mittal — Fri, 07 Aug 2026 13:29:56 +0000

Update, 8 Aug 2026

The corrected Qwen estimate below remains valid for this frozen model and workload,
but it is not a portable optimizer result. A fresh Llama replication reversed
direction. After an external review found a numeric-language mismatch in that
Llama control, I ran one preregistered exact canonical correction. The canonical
string control still scored 92/150 versus 82/150 for the immutable integer
treatment, with a paired interval of [-12.7, -0.7] points. See the complete
correction decision.

I had a result I wanted to believe.

Changing the model-facing JSON answer from a signed numeric string to a native
integer, then deterministically converting it back to the caller's string contract,
appeared to recover 14.3 percentage points of constrained accuracy on Qwen2.5-7B.

Then an audit found a real problem in the experiment runner.

The historical Outlines path could receive a prompt after the Qwen chat template had
already been applied, while the Outlines Transformers adapter could apply that
template again. The accepted artifacts remained complete and provenance-valid, but
the control and treatment evidence could no longer support an architecture decision
without a corrected paired run.

So I froze a new protocol, corrected both sides of the comparison, ran 200 new cloud
generations, downloaded every row, and validated the bundle independently.

The effect survived in direction and practical size:

Contract-valid correctness increased from 18/49 to 24/49, a paired gain of 12.2
percentage points, while final external-schema validity remained 100%.

But the uncertainty matters. The exact bootstrap interval is [0.0, 26.5] points
and exact McNemar p = 0.146. This is evidence for a scoped next stage, not proof of
a general solution.

The complete code, protocol, raw rows, hashes, traces, validation report, and decision
are public in the repository.

Vaibhav701161 / constrained-decoding-lab

Constrained Decoding Under Matched Conditions

A controlled, artifact-validated study of how JSON prompting, grammar-constrained decoding, and output-field order affect mathematical accuracy and schema compliance.

Progress presentation

Watch this presentation for an overview of the project's progress, key findings and results to date.

Watch on YouTube

Central result

This is not a claim that constrained decoding is universally harmful. It is a controlled…

View on GitHub

The system boundary I am testing

Suppose an API requires this external object:

{
  "reasoning": "...",
  "answer": "18000"
}

The answer must be a canonical signed-integer string. The model, however, may be
better aligned with a native JSON integer:

{
  "reasoning": "...",
  "answer": 18000
}

The proposed compiler does not weaken the caller's contract. It creates a separate
model-facing contract, generates once under that grammar, applies an exact inverse
transformation, and validates the final object against the original schema.

external schema
     |
     v
contract IR and applicability checks
     |
     v
model-facing schema plus replayable transform plan
     |
     v
one constrained generation
     |
     v
deterministic reverse transduction
     |
     v
original external-schema validation

For this experiment, the only semantic transform is:

external canonical signed-integer string
    -> internal JSON integer
    -> arbitrary-precision base-10 stringification
    -> external validation

There is no second model call, sign repair, rounding, regex guessing, or fallback
coercion. A boolean is rejected even though Python treats it as an integer subtype.
Lexical values such as "+26", "00026", "26.0", and "2.6e1" are not guessed
into the accepted language. Unsupported cases fail closed.

What the audit found

The audit identified three separate defects or inconsistencies.

1. A double-chat-template risk

For string inputs, Outlines owns chat templating through its Transformers adapter.
The runner had passed an already formatted chat string into that layer. A Qwen prompt
could therefore be wrapped twice.

The corrected path passes the raw project prompt to Outlines. Direct Transformers
and XGrammar generation use the same shared formatting helper exactly once. Tests
compare effective token IDs and prove that the nested form is different.

2. Backend-dependent generated-token counts

Direct generation and XGrammar counted generated tensor IDs. Outlines counted the
visible text after decoding. A backend-only stop token could make those metrics
incomparable.

The corrected metric retokenizes visible generated content without special tokens
for every backend. This is a measurement fix, not a model behavior change.

3. Noncanonical whitespace policies

Permissive JSON whitespace loops add legal token paths and can create avoidable
stalls. The corrected runner pins compact separators in XGrammar and an empty
whitespace pattern in Outlines.

Real pinned-backend tests verify that compact valid JSON is accepted and a long
whitespace prefix is rejected.

These corrections changed the experimental system. Comparing a corrected treatment
to a historical control would mix runner versions, so I reran both representations.

The protocol was frozen before the cloud run

The corrected question was narrow:

Under greedy FP32 decoding on Qwen2.5-7B-Instruct, does a model-facing JSON integer
improve contract-valid correctness over a signed numeric string when both sides use
the same corrected source and the final object must satisfy the unchanged external
contract?

The frozen design used:

Component	Value
Model	Qwen2.5-7B-Instruct
Dataset	Deterministic GSM8K-50 test subset, seed 0
Clean paired set	49 items after one previously documented contradictory reference exclusion
Decoding	Greedy, seed 0, FP32, 256 maximum new tokens
Backends	Outlines 1.3.2 and XGrammar 0.2.3
External contract	Reasoning string followed by signed numeric answer string
Treatment	Internal integer followed by deterministic stringification
Primary metric	Paired delta in contract-valid correctness

The job first ran the same five deterministic items in all four conditions. Expansion
to 50 was allowed only if row identity, prompt equivalence, schema validity, run
signatures, nonblank output, and cap status passed. The canary reused the same result
files, so it did not create duplicate research rows.

The Green rule required at least five points of recovery, 100% final external
validity, paired wins exceeding losses, no systematic replacement failure, and
evidence consistent with a representation-boundary mechanism.

Importantly, semantic correctness was not part of the canary. Stopping or expanding
based on the first five answers would have biased the confirmation.

The corrected result

The cloud job produced all 200 expected rows:

2 backends x 2 representations x 50 items = 200 generations

The clean paired analysis retained 49 items under the audit policy.

Representation	Correct	Contract-valid accuracy	Final external validity	Negative answers
Signed numeric string	18/49	36.7%	49/49	2
Internal integer plus transducer	24/49	49.0%	49/49	0

Paired effect:

net improvement:                  6 items
accuracy delta:                  +12.2 percentage points
exact bootstrap 95% interval:    [0.0, 26.5] points
treatment-only repairs:           9
control-only regressions:         3
both correct:                    15
both wrong:                      22
exact two-sided McNemar p:        0.145996

The intervention repaired three times as many items as it broke. It also preserved
the external contract for every clean treatment output.

The point estimate clears the preregistered continuation threshold. It does not
clear a conventional p < 0.05 threshold, and the interval touches zero. The honest
conclusion is not "the compiler works universally." It is "the corrected evidence is
strong enough to justify one bounded external-validity test."

Every item remains inspectable

Aggregate accuracy can hide cherry-picked wins or a concentrated replacement
failure. The item map below shows both outcomes for every clean-analysis item in the
frozen dataset order.

The nine repaired items were:

gsm8k_test_98
gsm8k_test_416
gsm8k_test_712
gsm8k_test_739
gsm8k_test_765
gsm8k_test_1205
gsm8k_test_1232
gsm8k_test_1251
gsm8k_test_1272

The three regressions were:

gsm8k_test_506
gsm8k_test_601
gsm8k_test_996

The regressions remain in the primary denominator. They are not treated as outliers.

The two backends became an implementation agreement check

Outlines and XGrammar emitted byte-identical raw output on all 50 signed-string items
and all 50 integer items.

That is strong evidence that both corrected integrations realized the same canonical
token path in this run. It also changes the statistical interpretation.

These are not two independent semantic replications. They are two implementations
agreeing on one effective 49-item paired experiment. Drawing separate effect bars and
calling the result replicated twice would overstate the evidence.

The observed semantic difference is associated with the model-facing representation,
not backend identity, under these matched settings.

The mechanism is subtler than "integers forbid negatives"

Both negative signed-string outputs were repaired by the integer treatment:

gsm8k_test_712:  -14 -> 14
gsm8k_test_1205:  -1 -> 4

But JSON integers allow negative values. The integer grammar did not prohibit the
minus token.

The XGrammar boundary traces show that the integer grammar removes quote and
structural-space continuations at the answer boundary while leaving minus legal. The
model therefore traverses a different legal token path. That path can affect the
first answer token and, because the answer follows reasoning, earlier generation as
well.

This is consistent with a representation-boundary mechanism. It is not proof that
every repair came from one local mask decision.

A correct benchmark answer can still have unfaithful reasoning

One repaired item is an important counterexample.

For gsm8k_test_712, the integer output's reasoning still computes -14, but its
final answer is 14. The benchmark score is correct. The generated explanation is
internally inconsistent.

So the evidence supports improved final-answer fidelity. It does not establish
improved reasoning faithfulness.

This distinction matters for any production system that exposes explanations,
executes intermediate values, or relies on chain-of-thought consistency.

The artifact validator does not trust the remote summary

After the Kaggle run completed, a separate local validator checked:

200/200 expected rows;
50 unique items in every condition;
source, runner, manifest, and result hashes;
run signatures and frozen item order;
prompt equivalence within each representation;
byte equality across the two backends;
generation errors, cap hits, and blank outputs;
internal and external schema validity;
the three preregistered boundary traces.

It reported zero failures and zero warnings. The run contained zero generation errors,
zero cap hits, zero blank outputs, and zero schema failures.

The summary was then regenerated locally. The paired bootstrap interval is computed
exactly from the empirical paired distribution by finite convolution, not estimated
from a random 10,000-resample loop.

What exists beyond the experiment

The repository now contains a conservative compiler prototype rather than only a
one-off transducer:

a canonical, hashable contract IR;
explicit unsupported-construct records;
deterministic, serializable alignment plans;
backend capability and whitespace requirements;
integer-string, key-alias, field-order, scratch-field, and whitespace transforms;
exact reverse transduction;
final validation against the original schema;
typed fail-closed refusals;
property and adversarial tests.

The implementation refuses references and schema features outside its initial
supported subset. That is intentional. A compiler that silently discards contract
semantics would be worse than no compiler.

The current test suite passes 87 tests, including 1,501 integer property cases and
real pinned-backend grammar probes. Those tests establish deterministic contract
behavior and integration. They do not substitute for external semantic evidence.

What changed relative to the historical result

The historical mixed-runner paths reported:

signed string:     61.2%
internal integer:  75.5%
delta:             +14.3 points

The corrected paired paths report:

signed string:     36.7%
internal integer:  49.0%
delta:             +12.2 points

The absolute rates changed substantially. The direction and approximate treatment
effect survived. Historical and corrected rates must not be pooled because prompt
handling and canonical grammar behavior differ between runner versions.

This is why preserving old artifacts matters. A correction should create a new
evidence layer, not rewrite what happened.

The decision: Green, but scoped

The corrected result satisfies the registered continuation rule:

The point recovery exceeds five percentage points.
Final external validity is 100%.
Paired wins exceed losses.
No systematic replacement failure was observed.
Boundary evidence remains consistent with the representation hypothesis.

The limitations are equally concrete:

only 49 clean paired items;
interval touching zero and p = 0.146;
treatment accuracy only 49.0%;
one model family and one benchmark;
one correct answer with inconsistent reasoning;
no independent semantic replication across backends.

The next authorized gate is not broad schema expansion. It is either:

the same paired contract test on a second model family; or
an executable tool-call task where contract-valid correctness has direct practical value.

If neither preserves a useful positive effect, the project should narrow into a
schema-risk linter and measurement system rather than continue claiming a general
compiler direction.

Reproduce the evidence

The corrected evidence lives under:

experiments/corrected-replication/

The main entry points are:

python scripts/validate_corrected_replication.py \
  --run-dir experiments/corrected-replication/results/qwen2.5-7b-corrected/results/corrected-replication \
  --dataset data/gsm8k_50_seed0.jsonl \
  --source-root deployment/kaggle/corrected-replication/source-snapshot \
  --kernel-source deployment/kaggle/corrected-replication/run_kaggle.py \
  --out /tmp/corrected-artifact-validation.json

python scripts/build_corrected_replication_figures.py

python -m pytest -q

Read the exact artifacts:

I am treating the correction as part of the result, not as an embarrassing footnote.
The point of an evidence pipeline is not to protect a claim. It is to make the claim
survive contact with its own implementation.

Constraints Cost 18 Points. Compiling the Schema Recovered 14.

Vaibhav Mittal — Wed, 05 Aug 2026 13:32:04 +0000

Update, 8 Aug 2026

This Qwen result remains valid for its frozen setup, but it did not generalize.
A fresh Llama replication was negative, and an external review later found that
the first Llama string control accepted a broader numeric language than the safe
compiler transform. I preregistered an exact canonical correction. The canonical
string control still scored 92/150 versus 82/150 for the frozen integer treatment,
with a paired interval of [-12.7, -0.7] points. The default optimizer thesis is
closed. See the complete correction decision.

My previous experiment ended with an uncomfortable result.

On the same 49 audited GSM8K questions, Qwen2.5-7B-Instruct answered 39 correctly
when prompted to produce JSON. When I enforced the declared schema with Outlines or
XGrammar, each backend answered only 30 correctly.

The constraints fixed compliance and cost 18.4 percentage points of recoverable
mathematical accuracy.

That result raised a more useful question than "are constraints bad?"

Was the model failing because it was constrained, or because the external contract
forced it through a poorly aligned lexical representation?

I have now completed the follow-up: 72 targeted and 150 full-confirmation generations
on Qwen2.5-7B, plus local integration probes, boundary traces, paired statistics, and
independent artifact validation.

The short result is this:

Replacing a model-facing signed numeric string with a native JSON integer, then
deterministically restoring the original string contract, improved both constrained
backends from 30/49 to 37/49 contract-valid correct.

That is a 14.3 percentage-point recovery while retaining 100% final external-schema
validity.

This article explains what changed, what stayed frozen, what the trace showed, and
why this is a Green light for a small contract-alignment compiler rather than proof of
a universal solution.

The first study is here:
Structured Output Fixed My JSON and Cut Math Accuracy by 18 Points.

All new code, raw rows, manifests, hashes, traces, and reports are in the repository.

Vaibhav701161 / constrained-decoding-lab

Constrained Decoding Under Matched Conditions

A controlled, artifact-validated study of how JSON prompting, grammar-constrained decoding, and output-field order affect mathematical accuracy and schema compliance.

Progress presentation

Watch this presentation for an overview of the project's progress, key findings and results to date.

Watch on YouTube

Central result

This is not a claim that constrained decoding is universally harmful. It is a controlled…

View on GitHub

The failure was more specific than "constraints hurt reasoning"

The external schema required this:

{
  "reasoning": "...",
  "answer": "18000"
}

The model naturally preferred this:

{
  "reasoning": "...",
  "answer": 18000
}

The prompt-only output was valid JSON and often mathematically correct, but it did
not satisfy the contract because answer was a JSON number rather than a string.

Hard constraints solved that type mismatch. They also changed the generated answer.

In the frozen reasoning-first baseline:

Condition	Semantic correctness	External-schema validity	Negative answers
Prompt-only JSON	39/49 (79.6%)	0/49	1/49
Outlines signed string	30/49 (61.2%)	49/49	12/49
XGrammar signed string	30/49 (61.2%)	49/49	12/49

Both constrained backends had the same paired transition against prompting:

correct in both:                  30
correct only with prompting:       9
correct only with constraints:     0
wrong in both:                    10
exact paired p:             0.003906

Eight of the nine losses were shared across Outlines and XGrammar. In seven shared
cases, the reasoning contained the correct positive magnitude and the final answer
field emitted its negative.

For example, the reasoning concluded 18000, then the constrained answer became
-18000.

That pattern suggested a representation problem at the answer boundary, not a reason
to build another grammar engine.

The hypothesis: compile the contract before generation

The caller's external contract remains authoritative. But the model does not
necessarily need to generate that exact wire representation directly.

The intervention was:

External contract
{"reasoning": "...", "answer": "18000"}

        compile
           ↓

Internal model-facing contract
{"reasoning": "...", "answer": 18000}

        constrained generation
           ↓

Deterministic transducer
integer 18000 -> canonical string "18000"

        external validation
           ↓

Returned contract
{"reasoning": "...", "answer": "18000"}

This design has four important properties:

It uses one model call.
It does not relax the external schema.
It does not use an LLM to repair another LLM's output.
It fails closed on ambiguous or unsupported values.

The core transducer is deliberately boring:

def canonical_integer_string(value: object) -> str:
    if isinstance(value, bool) or not isinstance(value, int):
        raise TypeError("internal answer must be a JSON integer and not a boolean")
    return str(value)


def transduce_integer_object(internal_value):
    if set(internal_value) != {"reasoning", "answer"}:
        raise ValueError("internal object must contain exactly reasoning and answer")
    if not isinstance(internal_value["reasoning"], str):
        raise TypeError("internal reasoning must be a string")
    return {
        "reasoning": internal_value["reasoning"],
        "answer": canonical_integer_string(internal_value["answer"]),
    }

After conversion, the rebuilt object is validated against the original signed
numeric-string schema. If parsing, typing, transformation, or external validation
fails, no external object is returned.

This is not post-hoc answer correction. The sign and magnitude generated by the
model are preserved exactly.

I wrote the decision rule before running the new matrix

The gate question was frozen before launching the 7B intervention:

Does a native JSON integer as the hard-constrained, model-facing answer
representation recover the semantic losses observed with a signed numeric string,
while deterministic stringification restores validity under the original external
contract?

The Green criteria were:

At least five percentage points of recovery over the matching constrained signed-string baseline.
A majority of the shared sign-loss cases repaired.
100% final external validity after transduction.
No new systematic semantic failure.

The model, dataset, item order, chat template, greedy decoding, seed, FP32 precision,
256-token cap, and backend versions remained frozen.

Relative to the previous prompt, the integer prompt changed only the symbolic answer
representation:

signed-string template: "answer": "<final numeric answer>"
integer template:       "answer": <integer>

A regression test compares the new prompt to the first accepted baseline row and
asserts that this replacement is the only change.

Layered scoring prevented a misleading result

The experiment reports four separate outcomes:

Semantic correctness: does the extracted numeric value match the gold answer?
Internal validity: did the model satisfy the model-facing integer schema?
External validity: did deterministic transduction produce an object satisfying the unchanged caller schema?
Contract-valid correctness: is the answer both semantically correct and externally valid?

This separation matters. A system that emits perfect JSON with the wrong number is
not successful. A mathematically correct response with the wrong API type is not
immediately usable either.

The primary product metric was contract-valid correctness after transduction.

First gate: an 18-item targeted suite

I mechanically derived the targeted suite from the frozen baseline artifacts. It
contained:

The union of all constrained semantic losses.
Matched controls where prompting, Outlines, and XGrammar were all correct.
Shared difficult cases where all conditions were wrong.
A recorded inclusion reason for every item.

The manifest recorded 9 Outlines losses, 9 XGrammar losses, 8 shared losses, and 10
unique losses across the two backends. The resulting suite had 18 items.

I ran four conditions:

Prompt-only integer.
Outlines integer.
XGrammar integer.
XGrammar unsigned numeric string, explicitly labeled diagnostic.

The unsigned-string condition was useful because every gold answer in this subset
was positive. But it was not contract-equivalent: the original external schema permits
negative integers, so an unsigned internal language cannot be the general solution.

Targeted result

Condition	Semantic correct	External valid
Prompted integer	13/18	18/18
Outlines integer	13/18	18/18
XGrammar integer	13/18	18/18
XGrammar unsigned-string diagnostic	13/18	18/18

The aggregate tie hid useful paired information:

Outlines integer repaired 7/8 shared signed-string losses.
XGrammar integer repaired 8/8.
Prompted integer, XGrammar integer, and XGrammar unsigned string had the same item-level correctness set.

The result cleared the preregistered threshold to advance. It did not count as final
confirmation because the suite was deliberately enriched for known failures.

What the answer-boundary trace showed

I wrapped the XGrammar Hugging Face logits processor and captured compact diagnostics
near the answer boundary. The trace records top pre-mask and post-mask candidates,
selected tokens, known sign and digit scores, and the number of masked vocabulary
entries. It does not store a full-vocabulary tensor.

Three items were traced:

gsm8k_test_173, a shared sign flip.
gsm8k_test_1216, a backend-difference case.
gsm8k_test_12, a matched control.

For gsm8k_test_173, the generated reasoning ended with the correct value 18000.
At the integer answer boundary after whitespace:

token "1" pre-mask score:  39.63
token "-" pre-mask score:  -1.33
selected token:             "1"

The integer grammar still permits legitimate negative values. The intervention did
not simply ban the minus sign. Instead, the changed representation exposed a token
path on which the digit was overwhelmingly preferred.

That is consistent with the hypothesis, but three traces are diagnostic evidence,
not a universal causal proof.

Full confirmation: 150 new 7B generations

After the targeted gate passed, I ran the three integer conditions on the complete
frozen 50-item subset:

50 prompt-only integer generations.
50 Outlines integer generations.
50 XGrammar integer generations.

One contradictory GSM8K reference remained in every raw artifact and was excluded
only from the predeclared cleaned analysis. No other row was removed.

Independent validation accepted all 150 rows:

50 unique item IDs in every condition.
Expected dataset, runner, schema, and source hashes.
No generation errors.
No token-cap hits.
100% final external validity.
Complete trace coverage.

The complete result

Condition	Semantic correct	Contract-valid correct	External valid	Negative answers
Prompted signed-string baseline	39/49 (79.6%)	0/49	0/49	1/49
Prompted integer + transducer	37/49 (75.5%)	37/49	49/49	0/49
Outlines signed-string baseline	30/49 (61.2%)	30/49	49/49	12/49
Outlines integer + transducer	37/49 (75.5%)	37/49	49/49	0/49
XGrammar signed-string baseline	30/49 (61.2%)	30/49	49/49	12/49
XGrammar integer + transducer	37/49 (75.5%)	37/49	49/49	0/49

The architecture figure is generated from the implemented path. The recovery figure
is generated directly from the accepted paired-summary.json; its counts and rates
are not manually typed into the image.

Both constrained backends gained 7 net correct answers, or 14.3 percentage points,
while preserving 100% external validity.

The original constrained gap was 18.4 points. The intervention recovered about 78%
of that gap.

The negative-answer cluster also disappeared in this matrix. Both constrained
backends moved from 12/49 negative answers to 0/49.

Paired effects, including the new failures

Aggregate improvement is not enough. A treatment can repair some items and silently
break others.

Outlines

signed-string correct:                 30/49
integer + transducer correct:          37/49
difference:                      +14.3 points
paired 95% interval:          [+4.1, +26.5]
newly correct / newly wrong:            8 / 1
two-sided exact paired p:              0.0391

XGrammar

signed-string correct:                 30/49
integer + transducer correct:          37/49
difference:                      +14.3 points
paired 95% interval:           [0.0, +28.6]
newly correct / newly wrong:           10 / 3
two-sided exact paired p:              0.0923

Outlines clears the conventional 0.05 threshold. XGrammar has the same point
estimate, but more discordant items, a wider interval touching zero, and a p-value
above 0.05.

The project threshold was not defined as "obtain p < 0.05 on every backend." It was
defined as meaningful recovery, majority repair, perfect final validity, and no
systematic replacement failure. Both backends passed that rule, but the uncertainty
around XGrammar belongs in the conclusion.

The incremental constraint cost disappeared under the integer representation

The cleanest comparison is not integer-constrained versus signed-string prompted.
Those conditions use different representations.

The matched comparison is integer-constrained versus integer-prompted.

All three integer conditions scored 37/49.

At the item level:

Outlines versus prompted integer had 3 treatment-only and 3 control-only correct items, for zero net difference.
XGrammar and prompted integer had the same correctness vector: 0 treatment-only and 0 control-only items.

So, in this experiment, I no longer observed an aggregate semantic tax from applying
the grammar after the representation was aligned.

That is the most important engineering result of the follow-up.

What did not work perfectly

The intervention did not restore every answer.

Prompted signed-string generation had 39/49 recoverable semantic correctness, while
all integer conditions had 37/49. The prompted representation difference was -4.1
points with a paired interval from -12.2 to +4.1 and exact p = 0.625.

The aligned systems also retained ordinary reasoning errors:

12/49 items were still wrong.
Outlines repaired 8 baseline errors and newly missed 1.
XGrammar repaired 10 and newly missed 3.

Representation alignment fixes a representation-associated failure. It does not
turn a 7B model into a perfect arithmetic solver.

An integration bug the local smoke test caught

The local Qwen2.5-0.5B smoke test exposed a separate XGrammar issue in my runner.

I initially reused one stateful Hugging Face logits processor across multiple
generations. The first item passed and later items failed with assertions. The fix was
to build a fresh processor for every generation.

The failed attempt is preserved as a diagnostic. The corrected five-item XGrammar
smoke passed 5/5 external validity, and the accepted cloud artifacts contain no such
errors.

This was a useful reminder: a structured-generation benchmark can be invalidated by
state management even when its schema and scoring logic are correct.

What this result proves

Within the declared setup, the evidence supports these statements:

The model-facing answer representation was materially associated with the constrained semantic loss.
A native integer plus deterministic external transduction recovered 14.3 points for both tested backends.
The original external signed-string contract remained 100% valid.
The observed negative-answer cluster disappeared.
Under the integer representation, neither backend had an aggregate accuracy loss relative to matched integer prompting.

What it does not prove

The evidence does not establish that:

Native integers improve every model or schema.
Every constrained-decoding failure is a representation problem.
The mechanism generalizes beyond Qwen2.5-7B, GSM8K, greedy FP32 decoding, and the tested library versions.
The result holds for legitimate negative gold answers. The transducer is tested on negatives, but this GSM8K subset contains positive targets.
The 49-item confirmation is an independent replication. The intervention was frozen before the new runs, but the underlying evaluation set also produced the original failure observation.
The complete contract compiler already exists.

The XGrammar confidence interval touching zero is another reason to keep the claim
narrow even though its point recovery matches Outlines.

The project decision

The gate is Green.

The result justifies building a bounded compiler layer with a small, auditable set of
safe transformations:

Canonical integer-string representation.
Field-order restoration.
Key aliases with exact inverse mapping.
Canonical whitespace policies.
Final validation against the unchanged external contract.
Explicit refusal when a transformation is ambiguous or unsupported.

The compiler should sit above existing engines such as Outlines and XGrammar. The
goal is not to compete with their grammar execution. It is to choose a safer internal
language for the model, prove that the transformation back is sound, and preserve the
caller's contract.

Before making a general product claim, the next evidence gates are:

Replication on an independent model family.
An executable tool-call task.
A fresh evaluation split.
Property tests over the supported schema subset.
Integration through a production inference path.

Inspect or reproduce the evidence

The repository contains the protocol, failure catalogue, schema variants,
transducer, tests, raw JSONL rows, manifests, compact traces, validation reports, and
paired summary:

The practical lesson is simple:

Do not assume the caller's wire format is the best language for the model.

Sometimes the safest way to preserve the external contract is to compile it into a
different internal representation, generate there, and transform back with code that
is deterministic enough to audit.

In this experiment, that small change recovered most of the lost accuracy without
weakening the guarantee.

Structured Output Fixed My JSON and Cut Math Accuracy by 18 Points

Vaibhav Mittal — Tue, 04 Aug 2026 10:43:51 +0000

I expected constrained decoding to change formatting.

I did not expect it to change which math problems the model could solve.

In a controlled Qwen2.5-7B experiment, both Outlines and XGrammar raised JSON Schema compliance from 0% to 100%. At the same time, recoverable mathematical accuracy fell from 79.6% to 61.2%.

That is an 18.4 percentage-point drop on the same questions, with the same prompt text, model, precision, decoding settings, and scoring code.

The paired result was even harder to dismiss:

30 questions were correct with both prompt-only and constrained generation.
9 were correct with prompting but wrong with constrained generation.
0 were rescued by constrained generation.
10 were wrong in both conditions.
Two-sided exact McNemar p = 0.003906.

This article explains how I reached that result, why "valid JSON" turned out to be an inadequate metric, and why moving one JSON field changed accuracy more than switching constraint backends.

All source code, raw outputs, validation reports, and accepted cloud runs are public.

Vaibhav701161 / constrained-decoding-lab

Constrained Decoding Under Matched Conditions

A controlled, artifact-validated study of how JSON prompting, grammar-constrained decoding, and output-field order affect mathematical accuracy and schema compliance.

Central result

This is not a claim that constrained decoding is universally harmful. It is a controlled reproduction showing that contract compliance and semantic correctness are separate outcomes, and that a decoder can improve the first while reducing the second under a specific, matched setup.

Principal results

The primary 7B…

View on GitHub

The engineering problem

Production applications rarely want an essay from an LLM. They want something a program can consume:

{
  "reasoning": "...",
  "answer": "42"
}

Prompting a model to return JSON is not a guarantee. It may add prose, omit a field, use the wrong type, or stop halfway through an object.

Constrained decoding addresses this at generation time. A grammar engine masks tokens that would make the output invalid, leaving the model only allowed continuations. Libraries such as Outlines and XGrammar make this practical.

The obvious expectation is:

Constrained decoding changes the syntax, not the answer.

But token masking changes the model's available next-token distribution. If reasoning and serialization happen in the same sequence, syntax and semantics may not remain independent.

That led to the question I wanted to test:

Can a decoder guarantee the output contract without changing the model's recoverable mathematical accuracy?

What I measured

I used a deterministic 50-question sample from the GSM8K test split and evaluated Qwen2.5-7B-Instruct under six conditions:

Free response.
Prompt-only JSON, reasoning first.
Outlines JSON, reasoning first.
XGrammar JSON, reasoning first.
Prompt-only JSON, answer first.
Outlines JSON, answer first.

The final matrix contains 300 validated generations.

One sampled GSM8K row had a contradictory reference answer. The question implies 240, while its stored reference is 150. I retained that row in the raw results, documented it in a machine-readable audit, and excluded it from the predeclared clean analysis. No other row was excluded.

The primary analysis therefore contains 49 paired questions per condition.

Controls held constant

Component	Setting
Model	Qwen2.5-7B-Instruct
Data	Deterministic GSM8K test subset, seed 0
Dataset SHA-256	`3639f2f6def0f50e02086bc91e6f4a45567c85aa9b0f498224cb9421400d812a`
Prompt formatting	`tokenizer.apply_chat_template(..., add_generation_prompt=True)`
Decoding	Greedy
Random seed	0
Maximum output	256 tokens
Precision	FP32
Constraint libraries	Outlines 1.3.2 and XGrammar 0.2.3
Primary sample	49 audited paired items per condition

For a matched comparison, the dataset item, prompt text, chat template, model, precision, decoding configuration, and scoring logic were held fixed. Only the treatment being tested changed.

Valid JSON is not the same as a valid contract

This distinction became central to the study.

I reported two correctness metrics:

Recoverable accuracy asks whether the intended numeric answer can be extracted from the response, even if the response violates the schema.

Strict accuracy requires the correct value to appear inside a schema-compliant answer field. This measures output that an application can use immediately without repair.

Consider this output:

{
  "reasoning": "40 + 2 = 42",
  "answer": 42
}

It is valid JSON. It is also mathematically correct. But it violates a schema requiring answer to be a numeric string:

{
  "reasoning": "40 + 2 = 42",
  "answer": "42"
}

The prompt-only reasoning-first condition exposed this exact failure mode:

100% valid JSON.
79.6% recoverable mathematical accuracy.
0% schema compliance.
0% strict accuracy.

The model consistently emitted an unquoted JSON number instead of the required string.

If I had measured only JSON parse success, I would have called this condition perfect. If I had measured only strict accuracy, I would have called every answer wrong. Both conclusions would have hidden important information.

The complete result

Condition	Recoverable accuracy	Strict accuracy	Schema compliance
Free response	36/49 (73.5%)	n/a	n/a
Prompted JSON, reasoning first	39/49 (79.6%)	0/49 (0.0%)	0.0%
Outlines, reasoning first	30/49 (61.2%)	30/49 (61.2%)	100%
XGrammar, reasoning first	30/49 (61.2%)	30/49 (61.2%)	100%
Prompted JSON, answer first	11/49 (22.4%)	8/49 (16.3%)	65.3%
Outlines, answer first	8/49 (16.3%)	8/49 (16.3%)	100%

Constrained decoding clearly solved the output-contract problem. Every Outlines and XGrammar reasoning-first response satisfied the schema.

But the recoverable view tells a second story. Relative to matched prompt-only JSON, Outlines and XGrammar each lost 18.4 percentage points of mathematical accuracy.

For each constraint backend:

accuracy difference: -18.4 percentage points
paired bootstrap 95% CI: [-30.6, -8.2]
two-sided exact McNemar p: 0.003906

Under the frozen strict metric, the interpretation reverses. Prompt-only JSON scored 0%, while both constrained backends scored 61.2%.

So these statements are simultaneously true:

Constraints greatly improved immediately usable correctness.
Constraints reduced recoverable mathematical correctness.

Schema compliance and semantic accuracy are separate outcomes. A structured-output evaluation should report both.

Aggregate accuracy was not enough

Because every condition used the same 49 questions, I could inspect correctness transitions item by item.

Against prompt-only reasoning-first generation, both constraint backends produced the same contingency table:

Paired outcome	Items
Correct in both	30
Correct only with prompting	9
Correct only with the constraint	0
Wrong in both	10

This is more informative than comparing 79.6% and 61.2% as independent proportions. The effect came from nine one-directional losses, not a balanced exchange of wins and losses.

Outlines and XGrammar also tied at 30/49, but they were not behaviorally identical:

29 items were correct under both.
One was correct only under Outlines.
One was correct only under XGrammar.
18 were wrong under both.
Only 20/49 raw responses were byte-identical.

An aggregate tie can conceal different generation paths.

Field order mattered more than backend choice

I initially treated JSON field order as a serialization detail. It was not.

The reasoning-first schema required:

{
  "reasoning": "...",
  "answer": "..."
}

The answer-first schema required:

{
  "answer": "...",
  "reasoning": "..."
}

Within each paired comparison, only that order changed.

For prompt-only JSON:

reasoning first: 79.6% recoverable accuracy
answer first:    22.4% recoverable accuracy
paired change:  -57.1 percentage points
95% CI:         [-71.4, -40.8]
exact p:         5.77e-8

For Outlines:

reasoning first: 61.2% recoverable accuracy
answer first:    16.3% recoverable accuracy
paired change:  -44.9 percentage points
95% CI:         [-59.2, -30.6]
exact p:         2.98e-6

Outlines maintained 100% schema compliance in both orders. Its 44.9-point decline therefore cannot be explained by an improvement in formatting.

In this setup, asking the model to commit to the answer before generating its reasoning changed task performance far more than switching between Outlines and XGrammar.

Trusting the GPU output required its own investigation

The final results ran on two Tesla T4 GPUs, but the first successful-looking runs were not necessarily trustworthy.

During the environment and precision diagnostics:

Some 4-bit and FP16 paths produced visibly corrupted tokens.
BF16 restored much of the structure but damaged digits in answers.
FP32 produced the first outputs accepted as task evidence.

These observations are specific to the tested Kaggle software and T4 hardware path. They do not establish that FP16 or BF16 universally fail for Qwen.

They do establish a broader engineering rule:

A completed GPU job is not automatically a valid experiment.

Before accepting a run, the validation pipeline checked:

Source-file hashes.
Dataset hash and exact item order.
Model artifact and environment.
Prompt and schema version.
Precision and decoding settings.
Expected row counts.
Duplicate item IDs.
Generation errors.
Token-cap hits.

Failed, corrupted, stale-deployment, and diagnostic runs remain in the repository instead of disappearing from the history.

What this result does not prove

This is not evidence that constrained decoding always harms reasoning.

The accepted evidence covers:

One deterministic GSM8K subset.
One final prompt family.
Two sizes from one model family.
Greedy decoding.
One primary precision.
Two grammar backends.
A sample of 49 audited paired questions.

The smaller Qwen2.5-0.5B experiment did not detect the same semantic constraint cost at its low base accuracy. That alone warns against turning the 7B observation into a universal rule.

The result also sits within existing research. JSONSchemaBench argues that structured generation should be evaluated across compliance, coverage, efficiency, and output quality. The XGrammar paper focuses on efficient grammar execution. CRANE explicitly studies how restrictive grammars can diminish reasoning and proposes reasoning-augmented constrained generation.

My contribution here is narrower: a reproducible engineering audit with matched prompt text, paired GSM8K scoring, two independent backends, field-order controls, strict versus recoverable metrics, and preserved failure evidence.

The next experiment I would run

The most useful follow-up is not another copy of the same matrix. It is a mechanism test.

I would separate reasoning from serialization:

problem
  -> unconstrained reasoning and answer
  -> constrained final JSON serialization
  -> schema-valid application output

That creates three pipelines to compare:

Prompt-only, single-stage generation.
Grammar-constrained, single-stage generation.
Unconstrained reasoning followed by constrained serialization.

If the two-stage approach preserves prompt-only accuracy while achieving 100% compliance, it would provide a practical way to keep the contract guarantee without applying a grammar across the reasoning trajectory.

After that, I would replicate on another model family, increase the preregistered sample, and add both a harder reasoning benchmark and a schema-centric benchmark.

Reproduce or inspect the evidence

The repository contains the deterministic dataset, runners, raw JSONL outputs, summaries, item-level reports, figures, environment records, and validation manifests:

The core lesson is simple:

Do not evaluate structured generation with one number.

Measure whether the output parses. Measure whether it satisfies the schema. Measure whether the answer remains correct. Then compare the same items, not just aggregate percentages.

Constrained decoding can solve the contract problem. That does not make its semantic effect zero.

Why “Return Valid JSON” Is Not a Decoding Constraint

Vaibhav Mittal — Sat, 01 Aug 2026 17:33:53 +0000

A model can know what valid JSON looks like and still produce a response that your parser rejects.

For example, each of these responses contains a valid-looking JSON object.

Here is the result:

```json
{"answer": "72"}
```

{"answer": "72"}

The calculation is 48 + 24.

{"answer": "72"}
{"answer": "72"}

A person can identify the answer immediately.

A program expecting exactly one JSON value cannot.

This is the distinction I wanted to understand more concretely:

Prompting changes which outputs are likely. Constrained decoding changes which outputs are allowed.

I built a small local lab around Qwen2.5-0.5B-Instruct to inspect that distinction at the tokenizer, generation, validation, and logit-processing levels.

The repository contains the scripts, raw generations, observations, and JSONL evaluation logs:

github.com/Vaibhav701161/constrained-decoding-lab

The experiment

I compared two generation conditions on three simple grade-school math problems.

The free-form prompt asked the model to show brief reasoning and end with a numeric answer.

The structured prompt asked it to return exactly one JSON object:

{
  "reasoning": "your reasoning here",
  "answer": "final numeric answer here"
}

Generation was deterministic, using:

do_sample=False

For every example, the evaluation script recorded:

the prompt and raw output
the parsed representation
JSON validity
schema validity
the extracted answer
exact-match correctness
prompt and generated token counts
latency
parsing or validation errors

The initial smoke-test result was:

Condition	Examples	Valid JSON	Correct
Free form	3	N/A	3/3
Prompted JSON	3	0/3	0/3

This should not be treated as a benchmark.

Three hand-selected examples are far too few, and the extraction logic is intentionally simple.

The failure modes were still useful.

The JSON-prompted outputs included:

Markdown-fenced JSON
valid-looking JSON followed by prose
multiple JSON objects in one response
continued generation after a complete object

In several cases, the answer was visibly present.

The complete response still failed JSON parsing.

That is the practical limitation of relying only on instructions such as:

Return ONLY valid JSON.
No markdown.
No extra text.

The instruction can make compliant output more probable.

It does not remove non-compliant output from the model’s possible decoding paths.

Validation happens after the mistake

A JSON parser can tell us that the output is invalid after generation.

A schema validator can tell us that the parsed object contains the wrong keys or value types.

Recovery logic may be able to extract the first complete object from a longer response.

These are all useful techniques, but they solve a different problem.

Post-generation validation asks:

Did the model produce something acceptable?

Constrained decoding asks:

Which tokens can the model legally produce next?

The first detects failure.

The second attempts to prevent structural failure.

This does not mean constrained decoding automatically makes an answer correct.

It only means the generated sequence is forced to remain inside a defined language.

A perfectly valid JSON object can still contain the wrong answer.

Structural validity and semantic correctness are separate evaluation dimensions.

Touching the decoding loop

To understand the underlying mechanism, I implemented a small Hugging Face LogitsProcessor.

The processor scans the tokenizer vocabulary and records token IDs whose individually decoded text contains a digit:

for token_id in range(len(tokenizer)):
    piece = tokenizer.decode(
        [token_id],
        skip_special_tokens=False,
    )

    if any(character.isdigit() for character in piece):
        self.banned_token_ids.append(token_id)

At every generation step, it modifies the scores of those tokens:

scores[:, self.banned_token_ids] = -float("inf")

After softmax, those tokens receive zero probability.

The decoder cannot select them.

This is not a grammar-constrained decoder.

It is a static token mask used to observe the basic decoding-time operation directly.

The result was more interesting than simply “the model stopped generating digits.”

It avoided ordinary ASCII digits, but produced digit-like Unicode characters such as circled numbers.

The formatting and output quality also degraded.

That exposed two important problems.

A constraint enforces its implementation, not its intention

“Prevent the model from outputting a number” is an informal intention.

“Set the logits of tokens whose individually decoded strings contain a digit to negative infinity” is an implementation.

Those are not equivalent.

A number can be represented through:

ASCII digits
other Unicode digit characters
number words
symbols that visually resemble numbers
token combinations missed by a naive vocabulary scan

The decoder only guarantees the rule that has actually been encoded.

This is especially important when people say that constrained decoding guarantees valid output.

The guarantee is relative to:

the grammar being enforced
the tokenizer representation
the parser state
byte and Unicode handling
the definition of an accepting state
stopping behaviour

A narrow or incorrect formalisation can provide a strong guarantee for the wrong language.

Hard constraints can damage output quality

The original model distribution may place most of its probability mass on a small set of natural continuations.

When those continuations are masked, the decoder must choose from what remains.

The remaining token may be structurally legal but awkward, unlikely, or semantically poor.

This is why evaluating constrained decoding only through a validity percentage is incomplete.

A useful comparison should also measure:

task accuracy
latency overhead
generated token count
dead ends
probability mass removed
output quality
recovery behaviour
behaviour across different schemas and prompt styles

A system can produce 100% syntactically valid JSON and still be worse for the actual task.

The real difficulty: grammars operate on bytes, models emit tokens

Setting selected logits to negative infinity is straightforward.

Determining which logits should be masked is the difficult part.

A grammar is usually defined over characters or bytes.

A language model emits tokens.

One token can contain:

one character
several characters
whitespace and punctuation together
a complete word
part of a multibyte character
a fragment whose behaviour depends on surrounding tokens

Suppose the parser currently allows a number beginning with 0.

The valid vocabulary might contain tokens representing:

0
0.
0.4
0.46
0}
0,

Whether each token is legal depends on the grammar and the current parser state.

It is not enough to ask whether a token equals the next valid character.

The decoder has to simulate consuming the token’s entire byte sequence.

Conceptually:

current parser state
        ↓
candidate token bytes
        ↓
simulate every parser transition
        ↓
keep the token or mask it

A token should remain available only when its complete byte sequence can be consumed without making the grammar invalid.

After one token is generated, the parser state changes.

The valid vocabulary set must therefore be computed again.

This makes a real constraint state-dependent.

A static digit ban uses the same mask at every generation step.

A JSON grammar needs a different mask after:

than after:

{"answer":

or:

{"answer":"72"

The legal continuation depends on where the parser currently is.

Why checking every token naively is expensive

A modern tokenizer may contain tens or hundreds of thousands of tokens.

Testing every token against the parser at every generation step would add substantial overhead.

This is where techniques such as token tries and cached parser transitions become useful.

A token trie groups vocabulary tokens by shared byte prefixes.

Instead of independently replaying every token through the parser, the decoder can traverse shared prefixes once and prune entire branches as soon as a prefix becomes invalid.

The high-level loop becomes:

Read the current parser state.
Traverse possible token-byte prefixes.
Prune branches that violate the grammar.
Collect token IDs ending on viable branches.
Mask the rest of the vocabulary.
Generate one token.
Advance the parser state.
Repeat until an accepting state is reached.

The implementation becomes more complicated for recursive structures, strings, escaping, numbers, UTF-8, and ambiguous parser states.

But the central mapping remains:

Parser state → legal token IDs

What this experiment does not prove

There are several limitations in the current setup.

The dataset is tiny

Three examples are enough for a smoke test, not for comparing decoding strategies.

The JSON extractor is deliberately strict and naive

It takes the content between the first opening brace and the final closing brace.

Multiple valid objects therefore become one invalid candidate.

A stronger evaluation should report separate metrics for:

whole-response validity
first-object recoverability
schema validity
semantic correctness

The free-form extractor is fragile

It selects the final number in the output.

Extra numeric text after the answer can produce an incorrect score.

The model was not prompted through its chat template

The raw prompt was passed directly to an instruction-tuned model.

Comparing this with the tokenizer’s intended chat template is necessary.

The token mask is not byte-accurate

Individually decoding each vocabulary token is useful for inspection.

A production implementation should reason carefully about the underlying token bytes and UTF-8 behaviour.

The constraint is static

The digit processor demonstrates logit masking, not parser-driven constrained decoding.

These limitations are not side notes.

They define what conclusions can and cannot be drawn from the experiment.

The next implementation step

The next useful milestone is a small state-dependent decoder for a restricted output language, for example:

{"answer":"<digits>"}

The implementation should:

define the language as an automaton or parser
map tokenizer IDs to byte sequences
track the current parser state
simulate candidate token bytes
allow only tokens that preserve validity
stop only in an accepting state
log mask size and latency at every step

Once that works, it becomes possible to compare:

prompting only
parsing and repair
regex or finite-state constraints
JSON-schema constraints
different token-filtering strategies

The comparison should keep the model, dataset, prompt content, and semantic scoring as controlled as possible.

The useful distinction

“Return only valid JSON” is still a useful instruction.

It can reduce failures, improve readability, and make the model’s intended output clearer.

But it is not a decoding constraint.

Prompting asks the model to follow a structure.

Validation checks whether it did.

Constrained decoding removes continuations that violate the structure.

These approaches can complement each other, but they do not provide the same guarantee.

The code, raw outputs, and notes for the experiment are available here:

Constrained Decoding Lab

I Expected JSON Grammar Masks to Kill Sampling Diversity. The Prompt Got There First.

Vaibhav Mittal — Thu, 30 Jul 2026 10:19:41 +0000

I started this project with a fairly simple suspicion.

When an LLM is forced to follow a JSON schema, the grammar removes most of
the vocabulary at every step. If I keep using ordinary sampling settings such
as temperature=0.7 and top_p=0.9, is there enough distribution left for
those settings to matter?

I expected the grammar mask to make the sampler act almost greedily. After
capturing the logits, checking the masks, and running 6,144 sampled outputs, I
got a less convenient answer.

For Qwen2.5-1.5B-Instruct, the sampler did collapse to one token very often,
but the grammar mask was not causing it in my main run. The model had already
made up its mind after seeing the full schema in the prompt.

A mask can remove 99% of token IDs while removing almost none of the
model's probability mass.

That sentence explained the first result. Then I withheld the schema from the
prompt and ran the comparison again. Legal probability mass fell from 0.9996
to 0.7962, and the mask started changing nuclei. The prompt had been doing
more work than I had given it credit for.

The short version: Qwen's grammar mask changed the nucleus size on 0 of
427 measured free-choice steps when the schema was shown. In a matched
follow-up with the schema withheld, it changed 120 of 556. TinyLlama's mask
changed 38 of 504 in the original run.

Measuring collapse

What I counted

Top-p sampling sorts tokens by probability and keeps the smallest prefix whose
total probability reaches p. Sampling then happens inside that prefix.

Suppose the next-token probabilities are:

token A: 0.93
token B: 0.04
token C: 0.02
token D: 0.01

At top_p=0.9, the nucleus contains only token A. The call is technically
sampling, but there is nothing to sample between.

I called that a collapsed step. More precisely:

collapse = nucleus size is 1

There is an important exception. If the grammar permits only one token, the
step is forced by syntax. A closing brace or quote should not count as evidence
that top-p removed diversity. I separated those steps before calculating any
headline rate.

The quantity I cared about was therefore:

collapsed free-choice steps
---------------------------
all free-choice steps

A free-choice step has at least two legal tokens before top-p is applied.

The same-logit comparison

The normal generation path is not enough for this question. A constrained
generation and a free generation can choose different tokens early, so their
later logits no longer describe the same prefix.

Instead, I captured one raw logit vector at each constrained-generation step
and evaluated two distributions from it:

the original model distribution
the same distribution after applying the grammar mask and renormalizing

For each one I recomputed nucleus size across a grid of temperatures and top-p
values. This gave me a same-logit counterfactual. I could ask whether the mask
changed the nucleus without confusing that effect with a different generated
trajectory.

I also recorded the legal probability mass:

Z_t = probability mass assigned to grammar-legal tokens

When Z_t is close to one, the model already wants to stay inside the grammar.
When it is small, the mask has to move a lot of probability.

The main run used:

Qwen2.5-1.5B-Instruct
TinyLlama-1.1B-Chat-v1.0
six JSON schemas
two prompts per schema
temperatures from 0.0 to 1.5
top-p values from 0.5 to 1.0

The schemas covered a tight enum, an integer, a free string, a patterned
string, a mixed object, and a mixed object with a reasoning string first.

One prompt detail is essential: _prompt_text appended the complete JSON
schema to the user message. Qwen saw the field names, types, enum values, and
required properties before I measured its legal probability mass. That setup
is common, but it is not the only way structured-output systems are built.

Checking the mask

Every result depends on the legal-token mask being right, so I started with a
schema small enough to check by hand:

{
  "type": "object",
  "properties": {
    "a": {"type": "integer"}
  },
  "required": ["a"]
}

I wrote a ten-state byte-level recognizer for the compact form {"a":12} and
compared its allowed tokens with XGrammar at every prefix. I also compared
XGrammar with Outlines.

The first comparison failed by exactly 271 token IDs at every prefix. That
number turned out to be the clue. Qwen's model head has 151,936 rows, while
its tokenizer has 151,665 entries. My checker was treating the padded model
rows as empty byte strings and accidentally allowing them as no-op tokens.

After excluding those rows, the hand-written recognizer, XGrammar, and
Outlines agreed at every checked prefix.

TinyLlama exposed another problem. SentencePiece can encode an opening brace
with a token whose grammar bytes include a leading space, even though the
ordinary decoded string looks like the same brace. The original hand check
was using the wrong beginning-of-sequence tokenization.

That bug was worth finding early. Later, the same leading-space behavior
created a misleading result when I compared generated token IDs with
re-encoded token IDs. All twelve TinyLlama sequences changed their first token
ID on the round trip, but the decoded bytes did not change at all. I now treat
the ID change as a tokenizer diagnostic, not as a changed output.

Results

Qwen: the two histograms are the same

With the full schema in the prompt, at T=0.7 and top_p=0.9, Qwen had 427
free-choice steps across 12 constrained generations. The nucleus contained
one token on 336 of those steps, a pooled rate of 78.7%.

The step count is not an independent sample size. Steps inside one generation
share a prompt, a schema, and a prefix. Giving each of the 12 generations
equal weight, the mean collapse rate was 82.5%, with a 10.0 percentage-point
standard deviation and a range from 70.0% to 100%.

The surprising part was the counterfactual:

Qwen measurement	result
pooled collapse rate	78.7%
mean legal probability mass	0.9984
mask changed nucleus size	0 of 427 steps
mask changed top token	0 of 427 steps

The free and constrained histograms sit directly on top of each other.

Figure 1. Free and constrained nucleus sizes at T=0.7 and top_p=0.9.
The two distributions overlap.

Qwen had the schema in its prompt and was already assigning almost all of its
probability to legal continuations. The mask removed vocabulary entries, but
hardly any probability. Removing low-probability tokens did not change the
top-p prefix.

The wider grid behaved in the expected direction. Lower temperature and lower
top-p produced more one-token nuclei. At top_p=1.0, there is no truncation,
so a free-choice step cannot count as collapsed under this definition.

Figure 2. Qwen collapse rate over the complete temperature and top-p grid.
At top_p=1.0, there is no nucleus truncation.

The prompt got there first

I ran a paired follow-up on the mixed-object schema. It includes strings, an
enum, an integer, a boolean, an array, and a nested object. I used one prompt
that still made sense without the schema:

Return a plausible record for a failed API request.

The model, grammar, sampler settings, and seeds 101 through 108 stayed fixed.
The only change was whether the full schema was appended to the user message.

measurement	schema shown	schema withheld
generations	8	8
prompt tokens	306	30
free-choice steps	543	556
mean generation `Z_t`	0.9996	0.7962
SD of generation `Z_t`	0.0001	0.0173
mask changed nucleus size	0	120
free counterfactual collapse	82.9%	59.5%
constrained collapse	82.9%	72.3%
mask-added collapse	0.0 points	12.8 points

Figure 3. Eight matched seeds on the mixed-object schema. Points in the left
panel are generation means; black markers show the mean and standard
deviation.

All 16 constrained generations finished and parsed as JSON. Every matched seed
had lower mean legal mass with the schema withheld. The mean paired drop was
0.2034. In the withheld condition, the mask changed 21.6% of free-choice
nuclei and changed the top token on 106 steps.

This is the mechanism I had missed. The mask's effect depends partly on how
much the prompt has already concentrated probability on legal tokens. It is
not determined by the model and grammar alone.

TinyLlama moved, but not by much

TinyLlama had 504 free-choice steps across 12 constrained generations. Its
pooled collapse rate was 76.2%. The generation-level mean was 76.7%, with an
11.6 percentage-point standard deviation and a range from 54.5% to 90.9%.

This time the mask had a measurable effect:

TinyLlama measurement	result
free counterfactual collapse rate	71.2%
constrained collapse rate	76.2%
mean legal probability mass	0.9239
mask changed nucleus size	38 of 504 steps
mask changed top token	36 of 504 steps

So the grammar added about five percentage points of collapse for TinyLlama.
That is real in this sample, but it is a much narrower result than saying
structured decoding generally turns top-p into greedy decoding.

The separate free control for TinyLlama was not clean enough to rescue a
larger claim. None of its 12 free generations parsed as JSON, and both prompts
hit the 128-token cap in four of the six schemas. The same-logit comparison is
the useful one here.

Schema shape still mattered

My original ordering prediction mostly survived. Tight enums and patterned
strings collapsed more often than ordinary string values. The gap was smaller
than I expected, and even free strings often had a one-token nucleus.

For Qwen at the reference setting, the schema-pooled rates ranged from 73.5%
for the free-string schema to 94.4% for the tight enum. TinyLlama ranged from
56.9% to 90.9% on those two schemas.

Figure 4. Qwen collapse rates by schema and approximate grammar position.

The position labels in this figure come from the decoded prefix. They are
useful for a rough breakdown, but a token that crosses two grammar positions
or an escaped string can fool the tagger.

Constraint pressure did not predict collapse

I expected low legal probability mass to line up with one-token nuclei. It
did not.

Pearson correlation between Z_t and the collapse flag was:

Qwen:      -0.020
TinyLlama: -0.056

Both are effectively zero in this sample.

Figure 5. Constraint pressure against the collapse flag for Qwen. The
correlation is -0.020.

The Qwen plot is mostly a pile of points near zero pressure because Z_t is
usually close to one. A few unusual steps do not explain the overall collapse
rate.

Whole-output diversity

The per-step measurement told me that the nucleus was often one token at
T=0.7, but it did not tell me whether sampler settings still changed whole
outputs.

For the second experiment I generated 6,144 Qwen outputs:

3 schemas
x 2 prompts
x 4 temperatures
x 2 top-p values
x 2 conditions
x 64 seeds
= 6,144 outputs

At top_p=0.9, the mean constrained distinct-output rate rose with
temperature:

temperature	constrained	free
0.3	14.3%	14.3%
0.7	42.4%	42.4%
1.0	49.5%	51.8%
1.5	60.7%	65.9%

Figure 6. Mean distinct-output rate across three schemas and two prompts,
with 64 seeds in each cell.

The constrained curve is not flat. Temperature plainly still did something.
For the free-string and mixed-object schemas, constrained and free distinct
counts matched in every top_p=0.9 cell on both prompts. The visible gap at
high temperature came mainly from the enum schema, where the grammar limits
the field to five values. That gap is the schema doing its job, not evidence
that the sampler settings had been disabled.

There is a length caveat at T=1.5. Some outputs hit the 128-token cap, and a
truncated string still counts as distinct. I would not treat the final points
as a clean measure of useful JSON diversity.

I also compared each step's probability distribution at T=0.7 and T=1.3
using total variation distance. Many steps barely moved because one token was
already dominant, while a smaller tail of steps remained temperature
sensitive. Once again, Qwen's free and constrained distributions looked
nearly the same.

Figure 7. Total variation distance between each Qwen step's distribution at
T=0.7 and T=1.3.

Order of operations can fail completely

I also checked where grammar masks sit in real sampling paths. In the pinned
versions I inspected, Hugging Face Transformers, vLLM, and SGLang all apply
the grammar mask before top-p.

For the Hugging Face path, the relevant order is:

penalties -> grammar mask -> temperature -> top-p

I deliberately reversed the last part and ran twelve seeds:

mask -> temperature -> top-p
temperature -> top-p -> mask

Mask-first completed all twelve generations with valid JSON. Top-p-first
failed all twelve because the top-p set and the legal-token set had an empty
intersection. Sampling then received an invalid probability row.

This was not a quality comparison. The reversed path simply failed. It also
confirmed that sampler order is part of the behavior, not an implementation
detail to ignore.

Implementation notes

I computed the metric grid while the full logit vector was still in memory.
My first storage plan saved only the top 4,096 logits and tried to reconstruct
everything later. That loses entropy information at high temperature and
cannot represent top_p=1.0, where the full support matters.

The first batched diversity run was also too slow because I moved full logits
to the CPU before sorting for top-p. Keeping logits, masks, and seeded
samplers on the GPU reduced the complete E2 run to 37 minutes and 24 seconds
with a batch size of eight.

I kept raw JSONL runs out of Git and committed the smaller aggregate CSVs,
figures, configs, source, and notes. The full repository is available here:

nucleus-collapse-constrained-decoding on GitHub

What this changes in practice

If the complete schema is already in the prompt, my Qwen results suggest that
a high one-token nucleus rate may be the model being confident, not the
grammar secretly turning sampling into greedy decoding. Temperature still
changed whole-output diversity in that setup.

If the schema is enforced by an API or gateway but withheld from the model,
the mask can do much more work. That is the setup where I would measure legal
probability mass and compare free and constrained nuclei on my own prompts
before assuming ordinary sampling settings behave the same way.

The practical question includes both the grammar engine and how much of the
output contract the model saw before decoding began.

What I would claim now

I would not say that JSON grammar masks generally collapse top-p sampling.
Two small models and twelve constrained generations per model are nowhere near
enough for that claim.

The prompt ablation is smaller still: one model, one schema, one prompt, and
eight matched seeds per condition. I read it as a mechanism check, not an
estimate of what every hidden-schema deployment will do.

What these runs do show is more specific:

A one-token nucleus can be common even after forced syntax steps are removed.
For Qwen2.5-1.5B-Instruct with the full schema in the prompt, the grammar mask did not cause that collapse at T=0.7 and top_p=0.9.
Withholding one mixed-object schema dropped mean generation Z_t from 0.9996 to 0.7962 and produced a 12.8-point mask effect on collapse.
TinyLlama showed a modest, model-dependent mask effect of about five percentage points.
Within the original schema-shown runs, constraint mass did not predict collapse in either model.
Temperature continued to change whole-output diversity in the larger Qwen run.

I began with a story about grammar masks silently disabling sampling. The
first measurements did not support it because the prompt had already pushed
Qwen onto legal tokens. When I removed that prompt information, the grammar
became visible in both probability mass and nucleus size.

That is not the result I planned to write up, but it is the one the logits
gave me.

Grammars are written in characters. Models emit tokens.

Vaibhav Mittal — Mon, 27 Jul 2026 13:44:55 +0000

I am about to spend several weeks building a constrained decoding engine, so I started by reading a paper from 2019.

Section 2.2 of the GPT-2 paper is about half a page long. It sits between the training dataset section and the model architecture section, and it is easy to skim past. It is also, as far as I can tell, the origin of the single hardest problem in the field I am about to work in.

Here is what it says, what I verified by running the tokenizer myself, and the thing I found at the end that I did not expect.

The part everybody already knows

A model computes on numbers. Text is characters. Something has to bridge that, and there are three obvious ways to do it.

Word level. Map every word to an id. The vocabulary explodes, because every inflection and typo and compound is a new entry, and anything the model never saw during training becomes <UNK>. The model goes blind on it.

Character level. Map every character to an id. Nothing is ever out of vocabulary, because you can spell anything. But sequences get roughly five times longer, transformer attention cost scales with the square of sequence length, and you have made the model learn spelling before it can learn meaning.

Subword level. Split into pieces. Common words stay whole, rare words break into fragments the model has seen in other contexts. Short sequences, no out of vocabulary hole.

Byte Pair Encoding is the subword method that won. It came from a 1994 data compression algorithm and got repurposed for NLP by Sennrich et al. in 2015. The training loop is four lines of pseudocode:

Start with a base vocabulary of every individual symbol in your corpus.
Count every adjacent pair of symbols.
Merge the most frequent pair into a new symbol and add it to the vocabulary.
Go back to step 2 until you hit your target size.

That is it. If t h shows up constantly, you get a th token. Then th e shows up constantly, so you get the. Run it 50,000 times and common English words are single tokens while rare ones stay in fragments.

The property worth holding onto: this is greedy, frequency based, and entirely dependent on the corpus. Nothing about the resulting vocabulary is linguistically principled. It is whatever compressed the training data. That is why different tokenizers split the same word differently and why numbers and code often tokenize in ways that look arbitrary.

What GPT-2 actually contributed

Standard BPE at the time ran over Unicode code points. Unicode has more than 130,000 of them, so your base vocabulary is enormous before you have merged anything, and you still have holes for anything outside your set.

GPT-2 runs BPE over UTF-8 bytes instead.

A byte has 256 possible values. That is your entire base vocabulary. And because every possible Unicode string encodes to some sequence of UTF-8 bytes, there is no out of vocabulary case. Not for any language, not for emoji, not for malformed input. Any string at all.

This is the actual point of byte level BPE and I think it usually gets described wrong. It is not primarily an efficiency win. It is a coverage guarantee bought with a tiny base vocabulary.

The paper is also honest about what broke when they first tried it. Applying BPE naively to bytes produced bad merges, because the greedy frequency heuristic learned separate tokens for dog, dog., dog!, and dog?. Four vocabulary slots for one word. So they blocked merges across character categories: letters do not merge with punctuation.

You can check that this worked:

'dog'   -> ['dog']         [9703]
'dog.'  -> ['dog', '.']    [9703, 13]
'dog!'  -> ['dog', '!']    [9703, 0]
'dog?'  -> ['dog', '?']    [9703, 30]

Same dog token, id 9703, in all four. The punctuation is separate every time.

Then they carved out an exception for spaces, because blocking every cross category merge fragments ordinary text badly. Spaces get attached to the token that follows them, not the one before:

'hello world' -> ['hello', 'Ġworld']

The Ġ is how GPT-2's vocabulary renders a leading space. It is a prefix, which means hello and hello are two different tokens with two different ids (31373 and 23748). The same five letters, in the same order, are a different symbol to the model depending on whether a space came first.

Hold that thought.

The trade nobody explains cleanly

Vocabulary size is a knob with a real cost on both sides.

Bigger vocabulary means shorter sequences for the same text, so fewer decode steps to say the same thing. But your embedding matrix and your final softmax layer both scale with vocabulary size, so you pay in parameters and in output projection compute.

Smaller vocabulary means a cheaper embedding and softmax, but longer sequences. And since decoding is one sequential forward pass per token, and decoding is the memory bound half of inference, more tokens is directly more wall clock time.

50,257 was GPT-2's answer. It is not a constant of nature.

Now the part I actually came for

Everything above is background. Here is why I was reading it.

Constrained decoding is the technique that forces a model's output to conform to a schema. At every generation step, before sampling, you set the logits of every token that would break the schema to negative infinity. After softmax those tokens have probability exactly zero. The model cannot emit them. Validity stops being something you hope for and becomes something structural.

To do that, you compile the schema into a state machine. At any state you know which characters are legal next, so you know which tokens are legal next, so you know what to mask.

Except the schema is defined over characters and the model emits tokens, and those two things do not line up.

I knew that abstractly before I started. What I did not appreciate is how badly they do not line up. So I ran GPT-2's tokenizer over some JSON.

import tiktoken
enc = tiktoken.get_encoding("gpt2")
print(enc.encode('{"'))        # [4895]
print(enc.encode('":'))        # [1298]
print(enc.encode('"}'))        # [20662]
print(enc.encode('"},{"'))     # [11919]

{" is one token. The opening brace and the opening quote of the first key are fused into a single symbol, id 4895.

": is one token. The closing quote of a key and the colon that follows it, fused, id 1298.

"},{" is one token. Id 11919. Five characters: close a string, close an object, comma, open an object, open a string. That is five separate transitions in any JSON grammar you would write, and the model emits all of them or none of them, as one indivisible unit.

A full tool call looks like this:

'{"location": "Jaipur", "unit": "celsius"}'

['{"', 'location', '":', 'Ġ"', 'Ja', 'ip', 'ur', '",',
 'Ġ"', 'unit', '":', 'Ġ"', 'cel', 's', 'ius', '"}']

Sixteen tokens. Look at where the boundaries fall. Not one of the structural delimiters sits alone. {", ":, ",, Ġ", "}. The grammar's alphabet and the model's alphabet are different alphabets, and the model's is coarser in exactly the places the grammar cares about most.

Out of 50,257 entries in the vocabulary, 50,001 are longer than a single character. That is 99.5 percent. 311 of them contain at least one of { } [ ] " : , and 92 of those also contain alphanumerics, which means there are almost a hundred single tokens that straddle the line between structure and content.

Why this costs you accuracy

Here is the mechanism, as I currently understand it. I want to be clear that this is the field's leading explanation rather than something settled, and testing it is a large part of what I am about to do.

Say a model is parsing a receipt and the correct total is 0.46. Left alone, it tokenizes like this:

'0.46' -> ['0', '.', '46']   [15, 13, 3510]

Three tokens, and 46 is a single one of them, id 3510. That is the path the model has walked thousands of times in training.

Now constrain it. The grammar is in some state where it expects a number, and depending on how your automaton is built and where the previous token landed you may end up masking in a way that forces the model down a different token path to spell the same string. The output is still valid JSON. It just took a route through the distribution that the model almost never took during training, in this context, and everything it generates after that point is conditioned on a prefix it did not choose.

BAML published a concrete case of this. On a receipt parsing task, constrained decoding returned 1 where the correct value was 0.46. Across the task they measured 91.37 percent accuracy constrained against 93.63 percent with free form parsing.

The 2 point gap is not the interesting part. The interesting part is that 1 is perfectly valid JSON. Your schema validated. Your types checked. Your pipeline reported success. There is no exception, no parse error, no signal anywhere that anything went wrong. The failure is silent by construction, because the entire purpose of the constraint was to make invalid output impossible, and it succeeded.

At the larger end, Tam et al. measured format restrictions degrading reasoning accuracy on benchmarks by as much as 27 points under strict constraints (EMNLP 2024). That number gets quoted a lot without its conditions attached, so treat it as an upper bound under specific settings rather than a headline, but the direction has since been replicated independently.

Token healing, and why it exists

There is a partial fix for the boundary half of this, and it has been sitting in Guidance for a while.

When a constraint truncates in the middle of what would naturally have been a single token, you back up to a clean token boundary and re-constrain from there, so the model gets to pick a token that spans the boundary rather than being forced to spell it out piece by piece.

It is called token healing, and reading GPT-2's section 2.2 is what made me understand why it needs to exist at all. It is not a workaround for a sloppy implementation. It is a direct consequence of a design decision made in 2019 for entirely unrelated reasons: bytes instead of code points, greedy frequency merges, spaces glued to the front of the following word. None of those choices were made with grammars in mind, because grammar constrained generation did not exist yet.

The XGrammar paper puts the same thing more precisely than I can. Their framing is that tokens are fixed strings which may not correspond to complete semantic units and may split Unicode characters, and that this is the fundamental challenge for structured generation. I found that sentence after I had worked out the problem by hand, which was a useful check that I was looking at the right thing.

What I do not know yet

Two things I want to measure rather than assume.

First, how much of the observed degradation is actually the boundary problem versus the other candidate mechanism, which is that a schema forces the model to commit to an answer field before the reasoning that would produce the answer has happened. My prior is that the second dominates on reasoning tasks and the first dominates on extraction tasks, but that is a guess and I have not seen it cleanly separated anywhere.

Second, whether the degradation is size dependent. If small models are hurt more than large ones, or the reverse, the shape of that curve says something about whether this is a capacity problem or a distributional one.

I am reproducing all of this across four model sizes over the next couple of weeks and publishing the harness with the numbers. If you have run into a case where structured output gave you a wrong but valid answer, I would like to see it. I am collecting them.

Reproduce it

Everything above runs in about thirty seconds:

# pip install tiktoken
import tiktoken
enc = tiktoken.get_encoding("gpt2")

def show(s):
    ids = enc.encode(s)
    toks = [enc.decode_single_token_bytes(i) for i in ids]
    print(f"{s!r:45} -> {toks}")

for s in ['dog', 'dog.', 'hello', ' hello',
          '{"', '":', '"}', '"},{"',
          '0.46', '46',
          '{"location": "Jaipur", "unit": "celsius"}']:
    show(s)

The GPT-2 paper is Language Models are Unsupervised Multitask Learners, Radford et al. 2019. Section 2.2 is the one to read, and it is shorter than this post.

Self-hosting SigNoz: notes from a day of actually running it

Vaibhav Mittal — Fri, 17 Jul 2026 17:33:33 +0000

Last week I self-hosted SigNoz, wired a small app into it, and spent a day poking at traces, logs, dashboards, and alerts. Most of it worked the way the docs said it would. Two things did not, and one of those turned into the most useful thing I learned all day: my logs and my traces were both arriving in SigNoz, but they didn't know about each other.

This post is a writeup of the whole thing, including the parts where I got stuck.

The app

I needed something real to observe, so I built Shelfie, a Flask + SQLite API for tracking books. It has the usual CRUD routes, plus a /books/<id>/recommendations endpoint that calls the Open Library API live. I also added two endpoints on purpose:

related-slow fetches all the book IDs for a subject, then queries each one individually (the classic N+1 pattern)
related-fast returns the same data in a single query

The idea was to have a known performance bug in the app and see how obvious SigNoz could make it.

Deploying SigNoz

SigNoz now ships an installer called Foundry:

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml

The casting.yaml is seven lines saying "docker compose, please". That generated and started six containers: ClickHouse, a Postgres metastore, the OTel collector, and the main signoz server.

Two things tripped me up here. First, the UI is on port 8080 now. A lot of older tutorials say 3301. Second, and this one cost me more time: the collector would not accept any telemetry at first. Its logs showed the opamp connection failing with "cannot create agent without orgId". It turns out that until you create the first admin account, there is no organization in the metastore, and the collector has nowhere to attach your data. I registered via the API (POST /api/v1/register) and the error went away immediately. So if your fresh SigNoz install seems to be dropping everything silently, check whether you ever finished signup.

Instrumenting Flask

Traces needed no code changes at all:

opentelemetry-instrument \
  --traces_exporter otlp --metrics_exporter otlp --logs_exporter otlp \
  flask run --port 5000

I added two custom metrics by hand (a counter for books created and a histogram timing the Open Library calls), then ran a small load generator against the app: normal browsing, recommendations, both related endpoints, and a steady trickle of requests for a book ID that doesn't exist, so there would be a real 404 rate.

One installation problem I would point out : I had pinned opentelemetry-distro==0.48b0, and opentelemetry-bootstrap then installed instrumentation packages from 0.65b0. Pip's resolver complained mid-install and the bootstrap failed. Upgrading everything to a matched set (SDK 1.44.0, instrumentation 0.65b0) fixed it. These packages really do need to move together.

A minute or so after starting the load generator, the Services page showed shelfie with a P99 of 1749 ms (the live Open Library calls are slow) and a 0.47% error rate from the deliberate 404s:

The N+1 in a waterfall

The slowest related-slow trace had 10 spans and took 74.27 ms: the Flask handler span, a connect, and then a stack of single-row SELECTs, each with its full SQL in the span attributes. The equivalent related-fast trace had 5 spans and took 7.94 ms. Same response body, about 9x the latency. You can see the problem in the shape of the waterfall before you read anything:

All of those spans came from auto-instrumentation (Flask, SQLAlchemy, requests). I wrote none of them.

The empty Logs tab

Now the part I actually want to tell you about. I opened one of those traces and clicked the Logs tab in the span details panel, expecting to see the request's log lines. Instead: "No logs found for this trace."

The logs were in SigNoz. I could query them in the Logs explorer. But their trace_id and span_id fields were empty, so nothing linked them to the trace. I checked ClickHouse directly to be sure:

trace_id:
span_id:
body:     127.0.0.1 - - [17/Jul/2026 13:28:33] "GET /books/subjects/romance/related-slow HTTP/1.1" 200 -

That log line is Werkzeug's access log, and it explains the whole thing. The access log is written after the request finishes. By that point the span has ended and there is no active trace context left to inject, so the exporter ships the line with no IDs. Everything was "working". The correlation just silently wasn't happening.

The fix was two small changes. I added a normal logger.info(...) call inside the recommendations handler, so at least one log line is emitted while the span is still open, and I set OTEL_PYTHON_LOG_CORRELATION=true so the logging instrumentation stamps the active trace context onto each record. After a restart, the new log line had a real trace_id in ClickHouse, and the same trace's Logs tab in the UI now showed it:

This trace-to-logs link ended up being my favorite feature in SigNoz. Being able to open a slow request and see only that request's log lines removes the usual step of grepping logs around a timestamp and hoping. But the lesson from the empty tab stands: a connected pipeline is not the same as correlated data. If your framework logs after the span closes, you get nothing, and no error tells you so. You have to open a trace and check.

A dashboard and a real alert

I built a small dashboard with two panels: the custom books-created counter, and a spans-per-minute panel filtered to service.name = 'shelfie'. One note on the counter: it's a monotonic sum, and with the default Rate aggregation my one burst of seed inserts was a barely visible blip. Switching the panel to Increase made it readable.

For alerting I wanted to see an actual delivery, not just a rule turning red. So I wrote a 15-line Python HTTP server that appends every POST body to a file, ran it on port 9999, and created a SigNoz Webhook notification channel pointing at it. One Docker detail: from inside the SigNoz container, localhost is the container, not my machine. docker inspect gave me the bridge gateway address (172.19.0.1), and I tested it with a wget from inside the container before configuring anything. The probe showed up in my receiver's log, so I knew the path worked.

Then I made a trace-based alert rule (span count for shelfie above 0 over a 5-minute rolling window, a threshold chosen to fire immediately under load) and waited. About ten minutes later my receiver logged this:

{"receiver":"shelfie-webhook","status":"firing",
 "alerts":[{"labels":{"alertname":"Shelfie traffic alert","severity":"critical"},
 "annotations":{"summary":"...(current value: 32) crosses the threshold (0)",
 "related_traces":"http://localhost:8080/traces-explorer?..."}}]}

And the rule showed Firing in the UI:

The related_traces field in the payload is a nice touch. It's a link that opens the traces explorer already filtered to the service and the exact time window that fired the alert.

Things I'd tell someone doing this tomorrow

Finish the signup before you debug ingestion. No account means no org, and no org means the collector rejects everything with only a vague log line to show for it.

The UI is on port 8080. Ignore anything that says 3301.

Log correlation is effectively opt-in. Auto-instrumentation gives you traces for free, but if all your logs come from an access logger that runs after the span ends, none of them will carry trace IDs. Log from inside the handler and set OTEL_PYTHON_LOG_CORRELATION=true, then verify by opening a trace's Logs tab. Don't just confirm that logs arrive.

Probe your alert channel before you create the rule. Testing the webhook path with one wget from inside the container meant that when the alert didn't arrive instantly, I knew the delay was the evaluation window and not Docker networking.

Keep the OpenTelemetry Python packages on matching versions. Mixed pins fail at install time in a confusing way.

All of this ran on one WSL2 machine in an afternoon: six containers, one Flask app, and a webhook receiver that now has a real alert payload sitting in its log file.

References: SigNoz docs, OpenTelemetry Python, SigNoz alerting