JSON Schema Doesn't Prevent AI Hallucinations (And That's Okay)

A few months ago, if you had asked me whether Structured Outputs solved hallucinations, I probably would have said yes.

Today, I wouldn't.

Not because Structured Outputs don't work-they absolutely do.

But because they solve a different problem.

After building ShapeCraft, an open-source structured output library supporting OpenAI, Groq, Ollama (GBNF), and Anthropic, I realized something that fundamentally changed how I think about production AI.

Structure and correctness are two completely different guarantees.

---

The misconception

Let's say you're extracting data from an invoice.

The invoice says:

Invoice Total: $900

Your model returns:

{
  "invoiceTotal": 1200
}

Let's validate it.

Validation	Result
Valid JSON	✅
Matches JSON Schema	✅
Required field present	✅
Correct data type	✅
Correct answer	❌

The response is structurally perfect.

The extracted value is still wrong.

JSON Schema didn't fail.

It did exactly what it was designed to do.

---

What JSON Schema actually guarantees

JSON Schema is responsible for ensuring your application receives data in a predictable format.

It guarantees things like:

• Valid JSON
• Required fields
• Correct data types
• Nested object structure
• Array validation
• Enum validation

That makes applications dramatically easier to build.

But it does not guarantee:

• The extracted value is correct
• The answer exists in the source document
• The model didn't hallucinate
• Business rules were followed

Those are different problems.

---

Structure ≠ Truth

I've started thinking about AI validation as two independent layers.

Layer 1 - Structural Validation

Can my application safely consume this response?

Examples:

• Is it valid JSON?
• Does it match my schema?
• Are required fields present?

JSON Schema solves this extremely well.

---

Layer 2 - Semantic Validation

Can I trust the information?

Questions become:

• Did the model extract the correct value?
• Can I trace it back to the source?
• Is there supporting evidence?
• Would I make a business decision based on this response?

This layer is much harder.

And I think it's where the next generation of AI tooling needs to evolve.

---

Different providers, different guarantees

Another interesting thing I learned while building ShapeCraft is that "structured output" isn't implemented the same way everywhere.

Provider	Implementation	Guarantee
OpenAI	Server-side schema enforcement	Native
Groq	Server-side schema enforcement	Native
Ollama	Grammar-constrained decoding (GBNF)	Constrained
Anthropic	Prompt + parsing + retries	Best-effort

Every provider can return valid structured output.

But they don't achieve it the same way.

Understanding those differences helps you make better architectural decisions.

---

What I changed in ShapeCraft

One design decision I made was to expose the guarantee level directly instead of hiding it.

const result = await generate(model, schema, prompt);

console.log(result.guaranteeLevel);

// native
// constrained
// best-effort

This doesn't tell you whether the answer is correct.

It tells you how confident you can be that the response structurally matches the schema.

I think that's an important distinction.

---

Where I think AI tooling is heading

Structured output solved one of the biggest problems in LLM development.

Before it, we spent hours fixing malformed JSON.

Today, that problem is largely solved.

The next challenge isn't generating valid JSON.

It's generating information we can trust.

Imagine every response including:

• Evidence for each extracted field
• Confidence scores
• Source attribution
• Semantic verification
• Reliability metrics

That's where I believe production AI is heading.

---

Final thoughts

I no longer think of AI validation as a single step.

It's two completely different layers.

First:

Is the response structurally correct?

Then:

Is the information actually correct?

Those are different questions.

And understanding that distinction has changed how I design AI systems.

I'm curious how you're handling this in production.

Do you rely solely on schema validation, or are you adding another verification layer after the model responds?

---

ShapeCraft

If you're interested in structured generation across multiple providers, ShapeCraft is open source.

It supports:

• OpenAI
• Groq
• Ollama (GBNF)
• Anthropic

With support for:

• Zod
• JSON Schema
• XML
• Regex
• Custom Validators
• Streaming
• Multi-turn structured conversations

GitHub:
https://github.com/aviasoletechnologies/shapecraft

Install

# npm
npm install @aviasole/shapecraft

# pnpm
pnpm add @aviasole/shapecraft

Provider SDKs:

npm install openai
npm install groq-sdk
npm install @anthropic-ai/sdk

# Ollama requires no additional SDK

Quick Start

import { generate, openai } from "@aviasole/shapecraft";
import { z } from "zod";

const schema = z.object({
  name: z.string(),
  age: z.number(),
});

const result = await generate(
  openai({ model: "gpt-4.1-mini" }),
  schema,
  "Generate a fictional user."
);

console.log(result.data);
console.log(result.guaranteeLevel);