The Parser Found a Verdict in the Model's Explanation

My review gate approved a diff the model had rejected.
The reject verdict was sitting right there in the output.
My parser grabbed an example block instead.

That is a twelve-line bug in my own code, not a model failure.
I ran the gate against MonkeyCode's free model access and its free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The short version

The loop is simple. Generate a patch, ask a model for a structured review, then accept or escalate.
Mine sent rejects to production as accepts.

The verdict came back with confidence 1.0 and an empty reasons list.
That pairing is impossible for a real review. It is very possible for an echoed schema example.

Two habits saved me. Log the raw model text next to the parsed object. Fail closed on any unexpected shape.

1. Symptom: high confidence, empty reasons

The log line looked healthy. verdict=accept confidence=1.0 reasons=[].

Ask yourself one question. Why would a careful review produce zero reasons?
It would not. Real accepts carry at least one justification, or a note that the diff is trivial.

So I dumped the raw completion for that run. The answer was sitting two lines above the real JSON.

2. What the model actually returned

My prompt had shown a schema example. The model echoed it, then answered.

The response schema is:
{"verdict": "accept", "reasons": [], "confidence": 1.0}

Review of the diff:
{"verdict": "reject", "reasons": ["drops the null check on line 42"], "confidence": 0.87}

Two JSON objects. The first is boilerplate I asked for. The second is the actual review.

Which one should a parser pick? The last one, always. Mine picked the first.

3. The parser that lied

Here is the code. It is short and it is wrong.

import json, re

JSON_RE = re.compile(r"\{.*?\}", re.S)  # first brace to first closing brace

def parse_verdict(text: str) -> dict:
    match = JSON_RE.search(text)          # search = first match wins
    if not match:
        raise ValueError("no JSON object found")
    return json.loads(match.group(0))

Non-greedy .*? plus search() means "first parseable object wins". On the output above, that object is the schema example.

The bug is silent. It parses cleanly, returns a dict, and looks healthy downstream.

Input shape	Old parser behaviour	Consequence
Example block first, verdict second	Returns the example	Wrong verdict, no error
Two verdict blocks, no example	Returns the first	Stale verdict wins
Truncated JSON, no closing fence	Raises on json.loads	Loud, acceptable
Verdict word inside reasons	Depends on the regex	Substring scans flip it

Look at the last row. If you "fix" this by scanning raw text for reject, you break the opposite way.

4. Fixtures that reproduce it

Never fix a parser without fixtures. These four cover the shapes I hit.

# test_gate.py
from gate import gate

EXAMPLE_FIRST = """The schema is:

json
{"verdict": "accept", "reasons": [], "confidence": 1.0}

Review:

json
{"verdict": "reject", "reasons": ["drops null check"], "confidence": 0.87}

TRUNCATED = '```

json\n{"verdict": "accept", "reasons": [], "confi'

NOT_FENCED = '{"verdict": "accept", "reasons": [], "confidence": 0.9}'

WORD_IN_REASONS = ('

```json\n{"verdict": "accept", "reasons": '
                   '["earlier \\"verdict\\": \\"reject\\" was wrong"], '
                   '"confidence": 0.9}\n```

')

def test_example_first_is_refused():
    assert gate(EXAMPLE_FIRST)[0] == "NEEDS_HUMAN"

def test_truncated_is_refused():
    assert gate(TRUNCATED)[0] == "NEEDS_HUMAN"

def test_unfenced_is_refused():
    assert gate(NOT_FENCED)[0] == "NEEDS_HUMAN"

def test_reasons_string_does_not_flip_verdict():
    status, verdict = gate(WORD_IN_REASONS)
    assert status == "ACCEPT"
    assert verdict["verdict"] == "accept"

Two blocks means refuse. That single rule kills the example echo.

5. The fix: one block, strict shape, fail closed

Change the contract first. Then change the parser.

Prompt rule: "Emit exactly one fenced json block. No examples. No prose outside the fence."

python
# gate.py
import json, re

FENCE_RE = re.compile(r"

```json\s*(?P<body>.*?)\s*```

", re.S)
KEYS = {"verdict", "reasons", "confidence"}
VERDICTS = {"accept", "reject"}

class VerdictError(Exception):
    pass

def parse_verdict_strict(text: str) -> dict:
    blocks = FENCE_RE.findall(text)
    if len(blocks) != 1:
        raise VerdictError(f"expected 1 fenced block, got {len(blocks)}")
    data = json.loads(blocks[0])
    if set(data) != KEYS:
        raise VerdictError(f"unexpected keys: {sorted(data)}")
    if data["verdict"] not in VERDICTS:
        raise VerdictError("verdict not in enum")
    reasons = data["reasons"]
    if not isinstance(reasons, list) or not all(isinstance(r, str) for r in reasons):
        raise VerdictError("reasons must be a list of strings")
    conf = data["confidence"]
    if not isinstance(conf, (int, float)) or isinstance(conf, bool):
        raise VerdictError("confidence must be a number")
    if not 0.0 <= conf <= 1.0:
        raise VerdictError("confidence out of range")
    return data

def gate(text: str) -> tuple[str, dict | None]:
    try:
        verdict = parse_verdict_strict(text)
    except (VerdictError, json.JSONDecodeError):
        return "NEEDS_HUMAN", None          # fail closed, always
    if verdict["verdict"] == "accept" and verdict["confidence"] >= 0.8:
        return "ACCEPT", verdict
    return "NEEDS_HUMAN", verdict

The 0.8 threshold is a policy choice, not a measured constant. Pick your own and write it down. Fail-closed is the part that matters. Unknown shape means a human looks, never a merge.

6. Where the free tier actually helped

I needed volume to find this bug. Adversarial fixtures are cheap to write and expensive to run.

MonkeyCode's free model access let me replay the fixture set every time I touched the prompt. The free server option let me run the gate as a throwaway job instead of wiring it into paid CI. Neither step needed a large model. A parser bug like this shows up on any model that echoes your examples.

The stated free allowance at the time of writing was 10M tokens. I will not promise that number holds. Quotas and server terms change, so read the project's current terms before planning around them. Check those terms before sending private source code to any hosted endpoint.

7. Limitations and who should skip this

Do not rely on a free hosted endpoint for anything with an SLA. Do not put secrets in prompts. Do not treat a model verdict as a substitute for your test suite.

If you need pinned model versions, predictable latency, or data residency, this workflow is not for you. Run the gate locally against your own endpoint instead.

Skip it too if nobody owns the escalation queue. A fail-closed gate is useless when NEEDS_HUMAN is a dead end.

8. A checklist you can steal

1. Log the raw model output beside the parsed object.
2. Require exactly one fenced JSON block, and count the blocks.
3. Validate keys, types, and enums before trusting a single field.
4. Fail closed on every unexpected shape.
5. Write one fixture per failure you have ever seen.
6. Test that a keyword inside a string cannot flip a verdict.
7. Hand-review NEEDS_HUMAN items within a day, or delete the gate.

The last item matters most. A gate nobody reads is just a log file with extra steps.

If you want to reproduce this, the four fixtures above are enough. Run them before you trust any review gate you own.