Structured Security Findings From a 7B Model: The Schema That Stopped Breaking

For about two weeks last year, roughly one in five runs of my audit pipeline died on the same line: JSON.parse. The model's security analysis was often good. The JSON wrapping it was a coin flip: a trailing comma here, a severity of "pretty high" there, one memorable case where the model wrapped valid JSON in a polite paragraph explaining that it had produced JSON.

I was asking a 7b model, qwen2.5-coder:7b running on Ollama, to emit security findings in a nested schema I'd designed the way I'd design an API response. That was the mistake. Here's the schema that eventually stopped breaking, and the loop around it that catches the rest.

Why nested schemas fail on small models

My first schema looked reasonable:

{
  "findings": [
    {
      "vulnerability": {
        "type": "reentrancy",
        "details": { "function": "withdraw", "lines": [42, 57] }
      },
      "assessment": {
        "severity": { "level": "high", "confidence": 0.8 },
        "exploitability": { "requires": ["external call"], "notes": "..." }
      },
      "remediation": { "suggestion": "...", "references": [] }
    }
  ]
}

A frontier model handles this fine. A 7b model fails it in ways that took me a while to categorize:

Every level of nesting is a place to lose track. The model is closing braces from memory. Three levels deep, mid-generation, with its attention mostly on the security reasoning, it forgets whether it's inside assessment or vulnerability. Frontier models have capacity to spare for bookkeeping. Small models spend theirs on the actual task, and the structure decays.

Optional fields invite improvisation. Give a small model a details object with loosely defined contents and it will invent keys, one per run, all different.

Mixed types compound errors. A confidence float next to a string next to an array of numbers means three formatting rules active at once. Each is easy. Together, at temperature, they're where the trailing commas came from.

The fix wasn't a better parser. It was accepting that schema complexity is a budget, and a 7b model has a small one.

The flat schema

Here's what I run now, essentially unchanged for months:

import { z } from "zod";

const Severity = z.enum(["critical", "high", "medium", "low", "info"]);

const Finding = z.object({
  title: z.string().min(1).max(120),
  severity: Severity,
  category: z.enum([
    "reentrancy",
    "access-control",
    "arithmetic",
    "unchecked-call",
    "oracle",
    "logic",
    "gas",
    "other",
  ]),
  location: z.string(),        // "withdraw(), lines 42-57" as plain text
  description: z.string(),
  recommendation: z.string(),
});

const FindingsResponse = z.object({
  findings: z.array(Finding),
});

Design rules that came out of the failures:

One level of nesting, total. An array of flat objects. The model never has to remember where it is.

Enums for anything you'll branch on. Severity as free text gave me "High", "HIGH", "high-ish", and "severe". Severity as a closed enum, with the options listed in the prompt, comes back clean almost every run, and when it doesn't, validation catches it instead of my sorting logic silently misfiling a critical.

Strings for anything descriptive. I originally wanted lines as an array of numbers. The model would emit ranges, or strings, or line numbers that didn't exist. Now location is prose and I treat it as a hint for a human, which is what it actually was all along. Parse precision out of model output only where you'll act on it programmatically.

No optional fields. Everything required, every run. Uniformity is legibility for a small model.

The severity enum earns a special mention because it's load-bearing: everything downstream in spectr-ai sorts and filters on it, so it's the one field where I accept zero creativity.

Validate at the boundary, and only there

Model output is untrusted input, the same as a network request. It crosses one boundary, gets parsed and validated once, and after that the rest of the codebase works with a typed value and never second-guesses it.

type Finding = z.infer<typeof Finding>;

function parseFindings(raw: string): z.SafeParseReturnType<unknown, { findings: Finding[] }> {
  // Small models love wrapping JSON in prose or markdown fences.
  // Extract the outermost JSON object before parsing.
  const start = raw.indexOf("{");
  const end = raw.lastIndexOf("}");
  if (start === -1 || end <= start) {
    return { success: false, error: new z.ZodError([]) } as never;
  }
  const candidate = raw.slice(start, end + 1);

  let json: unknown;
  try {
    json = JSON.parse(candidate);
  } catch {
    return FindingsResponse.safeParse(undefined); // uniform failure shape
  }
  return FindingsResponse.safeParse(json);
}

The slice-to-braces trick is inelegant and it eliminates the entire "here is your JSON:" class of failure in one move. I've made peace with it.

The retry loop that feeds errors back

The piece that took reliability from good to boring: when validation fails, don't just retry, tell the model what it got wrong.

async function getFindings(code: string, maxAttempts = 3): Promise<Finding[]> {
  let feedback = "";

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const raw = await ollamaGenerate({
      model: "qwen2.5-coder:7b",
      prompt: buildPrompt(code) + feedback,
      format: "json",
      options: { temperature: 0.1 },
    });

    const result = parseFindings(raw);
    if (result.success) return result.data.findings;

    const issues = result.error.issues
      .map((i) => `- ${i.path.join(".")}: ${i.message}`)
      .join("\n");

    feedback =
      `\n\nYour previous response failed validation:\n${issues}\n` +
      `Respond with ONLY a JSON object matching the schema. No prose.`;
  }

  throw new Error(`Model output failed validation after ${maxAttempts} attempts`);
}

Details that matter in practice:

Zod's error messages are the feedback. "severity: Invalid enum value. Expected 'critical' | 'high' | ..." is exactly the correction a model needs, and I get it for free from the validator I already run. No separate error-explaining code to maintain.

Low temperature for extraction. Creativity belongs in the analysis prompt, not the formatting one. Near-zero temperature cut my malformed-output rate noticeably on its own.

Ollama's format: "json" helps but doesn't save you. It constrains output to valid JSON syntax, which kills trailing commas, but valid JSON with a wrong shape or a hallucinated severity still gets through. Syntax enforcement and schema validation solve different layers, you want both.

Three attempts, then fail loudly. Early on I had a fallback that logged a warning and returned an empty findings array. That's the worst possible behavior for a security tool: a parse failure quietly became "no vulnerabilities found." If the pipeline can't produce validated output, it should say so at full volume.

On my machine, with all of this in place, first-attempt success is the strong norm and the retry loop handles nearly everything else. Parse failures went from a daily annoyance to something I check the logs for out of curiosity.

The general lesson transfers beyond security findings: with small models, don't fight for the output format you want, negotiate down to the format they can reliably produce, then enforce it mechanically at the boundary.

What's the most cursed thing a model has handed your JSON parser?