I was extracting customer details from onboarding messages. The result needed a name, an age, and a country code before it could be passed to the account service.
The schema looked strict enough:
const PersonJsonSchema = {
type: "object",
required: ["name", "age", "country"],
properties: {
name: { type: "string", minLength: 3 },
age: { type: "number", minimum: 18, maximum: 120 },
country: { type: "string", pattern: "^[A-Z]{2}$" },
},
};
Then a test message containing a 12-year-old came back as:
{ "name": "Jo", "age": 12, "country": "IN" }
It was valid JSON. It had every required field. Every field had the right basic type. It still should not have reached the account service.
My first fix was another instruction in the prompt
I added a sentence telling the model that age must be between 18 and 120, names must have at least three characters, and country codes must be two uppercase letters.
That improved the happy path. It did not give me a guarantee. A model can follow a constraint, forget a constraint, or confidently return a value that only looks plausible. Adding more prose to the prompt just made the instruction longer.
I was already using shapecraft to parse and validate the response, so I assumed the minimum and pattern keywords were supposed to be enforced there. When they were not, I briefly treated it as a bug.
It was not a bug. The built-in JSON Schema checker is intentionally a useful subset. It checks types, enum membership, required fields, nested properties, and array items. It does not pretend to implement every JSON Schema keyword.
That was actually a better design boundary than silently having each backend behave differently. I needed a full validator, not a larger prompt.
The strict validator belongs at the existing boundary
I installed AJV for the JSON Schema keywords I needed:
npm install ajv
Then I compiled the same schema once and passed a function to generate():
import Ajv from "ajv";
import { generate, groq } from "@aviasole/shapecraft";
const ajv = new Ajv({ allErrors: true });
const validatePerson = ajv.compile(PersonJsonSchema);
const result = await generate(
groq({ model: "llama-3.3-70b-versatile" }),
{ jsonSchema: PersonJsonSchema },
message,
{
maxRetries: 2,
jsonSchemaValidator: (value) => {
if (!validatePerson(value)) {
throw new Error(ajv.errorsText(validatePerson.errors));
}
},
}
);
The callback gets the parsed value. It returns normally when the value passes and throws when it does not. Shapecraft wraps that failure as a SchemaViolationError, so the normal retry behavior still applies. I did not need a second extraction loop, a second parser, or provider-specific code.
With the test message for a 12-year-old, both attempts failed the same minimum: 18 check and the call ended with MaxRetriesExceededError. With a valid message, the result came back as the same typed object I was already expecting.
The part I almost got wrong with oneOf
The next schema used two allowed identity shapes: either a company account with registrationId, or an individual account with dateOfBirth.
const AccountJsonSchema = {
oneOf: [
{
type: "object",
required: ["kind", "registrationId"],
properties: {
kind: { const: "company" },
registrationId: { type: "string", minLength: 5 },
},
},
{
type: "object",
required: ["kind", "dateOfBirth"],
properties: {
kind: { const: "individual" },
dateOfBirth: { type: "string", pattern: "^\\d{4}-\\d{2}-\\d{2}$" },
},
},
],
};
The built-in checker cannot express that kind of union. AJV can, but the important thing is that I still own the schema and the validator as a pair. The model generates toward the schema-derived prompt, and the validator is the final authority on whether the parsed value can leave the boundary.
What this does and does not change
The custom validator only changes validation for { jsonSchema } inputs. Zod schemas continue through Zod. Regex, XML, and GBNF inputs keep their own validation paths.
It also does not make the model's answer true. AJV can tell me that age is an integer in the allowed range or that an object matches one branch of oneOf. It cannot tell me whether the person really is that age. Structural correctness and factual correctness are still separate checks.
I kept the custom validator on the few boundaries that needed full JSON Schema semantics instead of replacing the default everywhere. That made the stricter behavior explicit at the call site and avoided accidentally changing older extraction paths.
Where it landed
The account service no longer receives a 12-year-old just because the JSON was well-formed. Values that violate minimum, maximum, minLength, pattern, or oneOf now fail through the same validation-and-retry path as any other schema violation.
The useful lesson was not "add more prompt instructions." It was to separate the model-facing schema from the validator implementation, then plug the validator that actually supports the rules the application depends on:
await generate(model, { jsonSchema: PersonJsonSchema }, message, {
jsonSchemaValidator: strictValidator,
});
If your JSON Schema contains keywords that a small built-in checker does not implement, you do not have to abandon the generation pipeline. Keep the schema, bring your validator, and let shapecraft keep the retry and backend plumbing around it.
Top comments (0)