DEV Community

Yusuf İhsan Görgel
Yusuf İhsan Görgel

Posted on

Check.isValidJson Failed on Correct JSON When the Model Wrapped It in a Markdown Fence

#dart #ai #llm #testing

llm_eval is a small Dart package for regression-testing LLM output: you write checks against a model's response, run them in CI, and read the outcome off two independent fields, passed and isError. Check.isValidJson is one of two checks shown in the README's Quick Start, and it exists for structured-output regression testing.

Until today's 0.3.1 release, that check called jsonDecode directly on the raw model output. When the model wrapped its JSON answer in a markdown code fence, which every mainstream chat-tuned model does by default, the check reported a failure even though the answer was correct and the JSON inside the fence was valid. The harness did not recognize the fence.

The repro

I ran this against a real model rather than a stub: ollama serve locally with llama3.2:3b, the same model test/ollama_e2e_test.dart already uses for end-to-end tests in this package. Prompt:

What is the capital of France? Answer with a JSON object containing a city field, formatted as a code block.
Enter fullscreen mode Exit fullscreen mode

Response:

```json
{
  "city": "Paris"
}
```
This is the capital of France.
Enter fullscreen mode Exit fullscreen mode

That is the right answer, and the JSON inside the fence is valid. I fed the exact string through the package at 0.3.0 as a path dependency, with no mocking involved:

passed: false
isError: false
detail: not valid JSON: Unexpected character
Enter fullscreen mode Exit fullscreen mode

Same result with the README's own quick-start example, verbatim:

Check.isValidJson(where: (v) => v is Map && v.containsKey('city'))
Enter fullscreen mode Exit fullscreen mode

against

```json
{"city": "Paris"}
```
Enter fullscreen mode Exit fullscreen mode

passed: false. The example in the README that is supposed to demonstrate the check did not work against a real model's default output format.

Flow through evaluate(): the raw jsonDecode runs first and succeeds directly for unfenced JSON, while fenced input throws a FormatException and falls through to the _fencedBlock regex, which extracts the inner content and decodes it into a pass carrying the detail 'parsed after stripping a markdown code fence'

Why isError: false is the part that matters

llm_eval keeps passed and isError as separate fields on purpose. The doc comment on CheckResult and the README both say the split exists so a broken harness is not mistaken for a failing model. A where callback that throws, for example, routes through describeError in lib/src/check.dart and comes back as an error result rather than a fail.

The fenced-JSON case is exactly the scenario that split is meant to catch, and it slipped through it. The harness was under-featured, and the result it reported, passed: false, isError: false, means the model got the answer wrong. The model had answered correctly; the check was the thing that failed.

For anyone running this in CI against a real endpoint, that is not a one-off. Every run where the model wraps its answer in a fence, the default for most chat models unless you force a JSON-only response mode, produces a false failure. That is noise, in a package whose reason for existing is separating real regressions from harness noise.

The maintainer already knew, informally

test/ollama_e2e_test.dart, line 79, prompts the model with:

...No prose, no code fences.
Enter fullscreen mode Exit fullscreen mode

That instruction is in the test suite because I had already hit this and worked around it by asking the model not to do the thing it does by default. It never made it into the library. Anyone using Check.isValidJson against a real model, without independently discovering and copying that same prompt wording, gets chronic spurious failures, with nothing in the output pointing at the prompt as the thing to change.

The fix

lib/src/check.dart, before:

CheckResult evaluate(String output) {
  Object? decoded;
  try {
    decoded = jsonDecode(output);
  } on FormatException catch (e) {
    return CheckResult.fail(detail: 'not valid JSON: ${e.message}');
  }
  final where = _where;
  if (where == null) return const CheckResult.pass();
  // ...
}
Enter fullscreen mode Exit fullscreen mode

After:

static final RegExp _fencedBlock = RegExp(
  r'```(?:json)?\s*\n?(.*?)\n?```',
  dotAll: true,
);

CheckResult evaluate(String output) {
  Object? decoded;
  var viaFence = false;
  try {
    decoded = jsonDecode(output);
  } on FormatException catch (e) {
    final match = _fencedBlock.firstMatch(output);
    if (match == null) {
      return CheckResult.fail(detail: 'not valid JSON: ${e.message}');
    }
    try {
      decoded = jsonDecode(match.group(1)!.trim());
      viaFence = true;
    } on FormatException catch (e2) {
      return CheckResult.fail(detail: 'not valid JSON: ${e2.message}');
    }
  }
  final passDetail = viaFence
      ? 'parsed after stripping a markdown code fence'
      : '';
  final where = _where;
  if (where == null) return CheckResult.pass(detail: passDetail);
  // ...where handling unchanged, now returns CheckResult.pass(detail: passDetail) on success
}
Enter fullscreen mode Exit fullscreen mode

The unfenced fast path is untouched: raw jsonDecode runs first, and correct unfenced JSON never reaches the regex. Only on a FormatException does it look for a single fenced block, optionally tagged json, with dotAll so the fence can span lines, and try decoding the captured content.

There is one genuinely new failure branch: a fence that matches but whose contents still do not parse, caught as e2. It returns the same not valid JSON: message shape as the original path, so the failure text a user sees is unchanged even though the control flow reaching it did not exist before.

The detail field on a pass reached through the fallback reads 'parsed after stripping a markdown code fence'. The fix avoids silently accepting fenced JSON and presenting the input as clean. A report from this path records which route decoded the value, so when you are working out why a check passed, you can see that it went through the fence-stripping branch.

Tests

test/check_test.dart now covers:

  • passes on a fenced block tagged json
  • passes on a plain fenced block with no language tag
  • passes on the exact repro shape: fenced JSON followed by trailing prose after the closing fence
  • confirms the where callback receives the value decoded from inside the fence rather than the raw string
  • fails when neither the raw string nor the fenced content parses as JSON
  • a regression guard: plain non-JSON prose with no fence still fails exactly as before

The fix adds a fallback path and leaves the definition of valid JSON alone. Prose with no fence and no JSON in it still fails the way it always did.

I also checked it the boring way: git stash on the fix file alone, rerun the repro, passed: false, the original bug. Restore the stash, rerun, passed: true. Full round trip, no drift in the diff.

What this means if you are writing similar checks

Check.isValidJson was written against the JSON spec: does this string parse as JSON. It was not written against how models that emit JSON actually respond when you do not constrain them, which is markdown-formatted prose with a fenced block inside it.

Any check you write against "the model's output" has both of those as inputs: the format your check expects, and the format the model tends to produce when you have not forced it otherwise. A README example that looks obviously correct can still fail the moment you point it at a real endpoint, because the gap sits in the shape of string that actually arrives rather than in the check's logic.

0.3.1 is on pub.dev now.

Top comments (0)