DEV Community: Yatin Davra

I Built a Slot-Filling Onboarding Bot. It Leaked a Validation Error Into the Chat

Yatin Davra — Tue, 21 Jul 2026 05:40:28 +0000

I was building an onboarding flow: a new user lands, a chat window asks them three things - what problem they're solving, what their idea is, who the user is - and once all three are answered, the app needs { problem, idea, user } as a clean, validated object. Not a transcript. Not "probably in there somewhere." An actual object my code could pass to the next step without re-reading prose.

My first pass was the obvious one: after every user reply, try to extract the object, check if it validated, and if it didn't, ask the model for a follow-up question. It worked right up until the extraction step threw partway through a conversation - a SchemaViolationError with actual validation details in it - and I hadn't wrapped that path carefully enough. The raw error text got forwarded straight into the chat window as if it were the bot's next question. A user saw "idea": expected string, received undefined where a friendly follow-up question should have been.

I fixed that specific leak with a try/catch. Then found a second one a day later in a different code path. That's when I stopped patching leaks one at a time and looked at whether the tool I was already using (shapecraft) had a mode built for exactly this shape of problem, instead of me re-deriving "never let internal errors reach the user" by hand, one bug at a time.

The actual design problem

It's not really "extract after every message and retry." It's two completely different concerns that I'd tangled into one loop:

Having a conversation - a model asking natural questions, probing vague answers, deciding when it has enough. That's chat. It should never be schema-constrained, because forcing JSON-shaped output mid-conversation is exactly how you get a bot that "helpfully" outputs a stray {...} into the chat.
Producing a validated object, once, at the end - a completely separate concern from the conversation itself.

Conflating them means every turn is a place validation could fail and leak. Separating them means there's exactly one place validation happens - after the conversation is over - and exactly one rule to enforce: nothing generated internally at that step is ever routed back into the chat.

`turnaround` mode

generate(..., { turnaround: true }) is built around that split. The model drives the conversation entirely through systemPrompt - what to ask, when to probe a vague answer, when it's satisfied. Shapecraft's only job during collection is to relay the model's reply and carry the transcript forward:

import { generate, anthropic } from "@aviasole/shapecraft";

const model = anthropic({ model: "claude-haiku-4-5-20251001" });

const FACILITATOR =
  "You are the onboarding facilitator. Ask exactly one focused question at a time to learn: " +
  "(1) what problem we're solving, (2) what the idea is, (3) who the user is. " +
  "Probe vague answers for specifics. Do not invent details on the user's behalf.";

let r = await generate(model, OnboardingSchema, "Hi", { systemPrompt: FACILITATOR }, { turnaround: true });
console.log(r.status === "collecting" ? r.message : r); // "What problem are you trying to solve?"

r = await generate(model, OnboardingSchema, "Onboarding takes too long", { systemPrompt: FACILITATOR }, {
  turnaround: true,
  memory: r.memory, // thread the running transcript back in
});
console.log(r.status === "collecting" ? r.message : r); // "What's your idea to fix that?"

// ...more turns, same pattern...

if (r.status === "complete") {
  console.log(r.data); // { problem, idea, user } - validated once, here, never mid-conversation
}

Notice FACILITATOR never mentions how to signal "done" - I don't have to write that part. Shapecraft appends its own completion instruction (plus a checklist built from the schema's required fields) to whatever system prompt I pass in, so the sentinel handshake isn't something I own or could get subtly wrong.

memory is a plain JSON-serializable object - just the transcript plus a turn count - so a stateless HTTP handler can persist it between requests and rehydrate it on the next one, which is the actual shape a real chat backend needs anyway.

The completion sentinel

The model signals "I'm done" by replying with a literal <<<COMPLETE>>> instead of another question - the instruction to do so is injected automatically, not something I write myself. That reply is the trigger for a single, separate generate() call - a normal structured extraction pass over the whole transcript, not part of the conversation the user sees. I liked this over having the model just emit JSON directly once it thinks it's done, for the same reason my original leak happened: raw or premature JSON has no business anywhere near the user-facing chat, and "is this the sentinel string" is a trivial check instead of a fragile "does this look like JSON" guess.

The rule that actually fixes my bug

Every response from a turnaround call is one of two shapes: { status: "collecting", message } where message is only ever the model's own conversational text, or { status: "complete", data } where data is only ever produced by the end-of-conversation extraction pass. There is no third path where something shapecraft generated internally ends up in message. If that end-of-conversation extraction fails validation, it throws a real, terminal error out of generate() - it does not get wrapped into a fake follow-up question, and the user is never re-interrogated for answers already sitting in the transcript.

That last part surprised me at first - I expected "validation failed, so ask again." But if the user's answers are already in the transcript, a validation failure at the end means my extraction step is broken, not that the user needs to repeat themselves. Re-asking would be asking the wrong party to fix the wrong problem.

Where that leaves it

My onboarding bot now has exactly one place structured data gets produced, and exactly zero places an internal error can masquerade as a chat message:

import { generate } from "@aviasole/shapecraft";

Repo's at github.com/aviasoletechnologies/shapecraft, package is @aviasole/shapecraft on npm. If you've ever had a slot-filling bot leak something internal into the chat window - curious how you caught it.

I Had a GBNF Grammar File and Nowhere to Point It. So I Added Support for It.

Yatin Davra — Fri, 17 Jul 2026 06:44:46 +0000

I was building a classification step that had to run entirely on a local model - no cloud calls, straight through llama.cpp. Read a chunk of text, output exactly one of a handful of fixed labels. It ran in a tight, low-latency loop with basically no retry budget, and local models - even good quantized ones - drift more on output formatting than the big hosted APIs do. Wrap the label in a sentence, add a stray quote, whatever. Small thing, but multiplied over a few thousand calls it's a real failure rate.

The fix already exists: llama.cpp grammars, better known as GBNF. Hand llama.cpp a grammar and it masks the token logits at every generation step, so the model literally cannot produce a token that would break the grammar. Not "probably won't" - cannot.

I already had a grammar file. What I didn't have was a way to hand it to the structured-output library I was using (shapecraft) - it had five schema-input types at the time (Zod, JSON Schema, regex, a validator function, XML), and none of them were built for "apply this formal grammar directly."

Why the existing schema types didn't fit

Not JSON Schema - GBNF describes a context-free grammar over an arbitrary string, not a JSON shape. My output was going to be a matched string, not an object.
Not a regex pattern - GBNF grammars are recursive (rules referencing rules, alternation, repetition). You can't flatten that into one flat regex.
Not a validator function - a validator checks output after generation. I wanted the model physically unable to produce the wrong shape, not a check that catches it afterward.

So I added a schema type that did fit.

What I built

A new schema-input type, { gbnf: grammarString }:

import { generate, llamaCpp } from "@aviasole/shapecraft";

const sentimentGrammar = `root ::= "positive" | "negative" | "neutral"`;
const local = llamaCpp({ modelPath: "./models/llama-3.2-3b-instruct.gguf" });

const result = await generate(local, { gbnf: sentimentGrammar }, "Classify: 'I love this!'");

console.log(result.data);           // "positive" - cannot be anything else
console.log(result.guaranteeLevel); // "constrained"

result.data is a raw string, not a parsed object - GBNF describes a string language, not a JSON shape.

A llamaCpp() backend (node-llama-cpp) - the thing that actually applies the grammar at the token level.

A bundled GBNF interpreter for every other backend (openai, groq, anthropic, ollama), none of which have a grammar parameter. There, the grammar gets injected into the prompt instead, and the returned string is checked against it - mismatch, retry:

import { generate, openai } from "@aviasole/shapecraft";

const date = await generate(openai({ model: "gpt-4o-mini" }), {
  gbnf: `root ::= year "-" month "-" day\nyear ::= [0-9]{4}\nmonth ::= [0-9]{2}\nday ::= [0-9]{2}`,
}, "When did WWII end in Europe?");

console.log(date.data);           // "1945-05-08"
console.log(date.guaranteeLevel); // "best-effort" - no grammar param on this API

Same grammar, same call, two very different guarantee levels depending on the backend. GBNF ended up being the one schema type where the guarantee genuinely depends on which backend is running it, not just retry count.

Then I tried to break my own validator

I don't trust a validator only tested against grammars that were supposed to pass.

Catastrophic backtracking - the classic thing that blows up naive regex engines, like ("a" "a"?)* "b" against a long run of as with no trailing b. Didn't blow up - milliseconds, every time. The matcher dedupes by position reached, not path taken, so it doesn't re-explore the same dead end repeatedly.

A genuinely adversarial grammar did slow it down - a ~26-way ambiguous alternation against a 300,000-character input with no valid terminator. That trips a step-budget guard in under a second with a clear error instead of hanging.

Deep right-recursion through a rule reference found a real limit. */+ repetition is iterative, no limit tested up to 50,000+ repeats. But a grammar like list ::= item "," list | item recurses through the JS call stack once per repetition, and breaks around 900-1000 reps. Instead of a raw stack-overflow trace, it now throws a clear message telling you to rewrite with */+ instead.

A real bug this surfaced

Stress-testing also found something unrelated but genuinely broken: the Groq backend was forcing response_format: json_object on every call, including GBNF ones. Groq's API rejects that mode if the prompt doesn't literally contain the word "json" - which a GBNF grammar prompt has no reason to. Every GBNF call against Groq was failing before the model even ran. Two-line fix, but only found by trying to break the feature instead of just checking the happy path.

Where that leaves it

From "I have a grammar file and nowhere to point it" to a schema type that's genuinely valid by construction on a local llama.cpp model, and checked (not just trusted) everywhere else - same call shape, same retry behavior as every other schema type.

If you've got a .gbnf grammar sitting around, or want a local model physically incapable of drifting off a fixed format:

import { generate, llamaCpp } from "@aviasole/shapecraft";

Repo's at github.com/aviasoletechnologies/shapecraft, package is @aviasole/shapecraft on npm. Curious if anyone else here is running GBNF grammars against local models, and what for - I only had the one use case in mind.

How I ended up contributing five ways to shape an LLM's output

Yatin Davra — Fri, 03 Jul 2026 06:46:44 +0000

A few months back I was on a project that needed LLMs to return actual structured data, not prose I'd have to regex my way through. Simple enough in theory, except every provider wanted the shape described differently - Anthropic's SDK wanted a tool schema, OpenAI had its own response-format, Ollama was basically raw prompting and hoping. I was rewriting the same schema three times per feature and writing my own retry-on-invalid-output loop for anything that didn't guarantee output.

That's how I found ShapeCraft (@aviasole/shapecraft on npm). Define a schema once, hand it a model, get back validated data plus a guaranteeLevel telling you how much to trust it - "guaranteed" means something different on OpenAI's native schema enforcement than on Anthropic's prompt-and-validate approach than on Ollama's token-level grammar constraints.

It already supported Zod out of the box:

import { z } from "zod";
import { generate, openai } from "@aviasole/shapecraft";

const schema = z.object({ name: z.string(), score: z.number() });
const result = await generate(openai({ model: "gpt-4o-mini" }), schema, prompt);

That covered most of what I needed. But a few things in my actual project didn't fit "define a Zod schema," so instead of working around it, I opened some PRs.

Part of the data I was extracting already had a JSON Schema, generated by another tool upstream. Rewriting it as Zod types just to satisfy the library felt like busywork, so that became the first addition - pass the JSON Schema straight through:

const result = await generate(model, {
  jsonSchema: {
    type: "object",
    properties: { name: { type: "string" }, score: { type: "number" } },
    required: ["name", "score"],
  },
}, prompt);

Then I hit cases where I didn't need an object at all, just a string in a specific format. Wrapping that in an object schema felt like putting a hat on a hat, so next came a plain regex option:

const result = await generate(model, {
  pattern: /^\d{4}-\d{2}-\d{2}$/,
}, "What is today's date?");

After that, a rule that only made sense as logic, not a type - something like "required only if this other field equals X." Technically expressible as a schema, miserable to read. So I added a way to pass a validator function directly, plus a hint so the model still has something to aim for:

const result = await generate(model, {
  validate: (output) => typeof output === "object" && output !== null && "id" in output,
  hint: { type: "object", properties: { id: { type: "string" } } },
}, prompt);

The one that mattered most to me was separate from this project entirely. I also do Tally/TDL and GST integration work, and that world runs on XML - nested tags, required fields buried a few levels deep. None of the above helped there since the output isn't JSON-shaped at all. So the last addition was template-based XML: give it an example with typed placeholders, model fills them in.

const result = await generate(model, {
  xml: {
    template: `<book>
  <title>{string}</title>
  <author>{string}</author>
  <year>{number}</year>
</book>`,
    required: ["title", "author"],
  },
}, 'Extract: "Clean Code" by Robert C. Martin, 2008.');

This one took the most back-and-forth. Placeholders are deliberately limited to {string}, {number}, {boolean}, and a typo throws before the model is even called - that's a template bug, not something worth a retry.

The trickier part was literal text. Anything outside the {} is supposed to survive untouched, but it's best-effort by default - a model will occasionally "improve" fixed text that reads like an instruction, since nothing marks it as untouchable. There's an enforceLiterals: true flag that force-corrects every literal after the fact if you need that guaranteed. Or, often simpler, just leave the fixed value out of the template and splice it into result.data yourself afterward.

required also checks non-emptiness at any depth, so <items></items> still triggers a retry instead of counting as present. And arrays plus parse: true gets you back a parsed JS object instead of raw XML, with specific nodes coerced into arrays even with a single item.

One caveat worth knowing up front: XML generation is prompt-driven on every backend, no token-level grammar constraint like Ollama has for JSON. It leans on the model being capable, especially with deeply nested templates.

All five of these route through the same generate() call underneath - same retries, same guaranteeLevel, same error types (SchemaViolationError, MaxRetriesExceededError). Different doors into the same room, which is probably why it was easy to keep adding them.

Genuinely curious about a few things: if you've built template-based XML generation before, does enforceLiterals feel like the right approach, or is there a cleaner way without a re-serialize pass? Is there a schema style still missing that you've actually needed in production - YAML, something protobuf-shaped? And for the validator path, is a hint object enough context, or would you want more?

Repo's here: github.com/aviasoletechnologies/shapecraft. Package is @aviasole/shapecraft. I'm using it across a few different projects now, so I'll keep adding to it as I run into more edge cases.

How to Create Multi-Page TIFF Files in Node.js (Without ImageMagick)

Yatin Davra — Tue, 14 Apr 2026 13:11:31 +0000

If you've ever tried to create a multi-page TIFF in Node.js, you know the pain. Most solutions require ImageMagick as a system dependency, which is a headache in Docker, serverless environments, or CI pipelines. Others wrap native binaries with brittle child_processcalls.

I ran into this exact problem while building a document processing pipeline — I needed to merge multiple scanned images into a single multi-page TIFF programmatically, with no external system dependencies.

After digging through the options, I ended up writing a pure Node.js solution and published it as an npm package: multi-page-tiff.

The Problem

Multi-page TIFF files store multiple images as linked Image File Directories (IFDs) inside a single .tiff file. Creating one correctly means writing the TIFF binary structure yourself — most image libraries only handle reading, not writing multi-page TIFFs.

Common workarounds people try:

ImageMagick via child_process— works but requires ImageMagick installed, breaks in serverless
sharp — great library but doesn't support writing multi-page TIFFs
tiff-multipage — last published 3 years ago, minimal maintenance
tiff-to-png — converts TIFF to PNG, not the other way

The Solution

npm install multi-page-tiff

const { imagesToTiff, imagesToTiffBuffer } = require('multi-page-tiff');

const fs = require('fs');

const images = [
  fs.readFileSync('./page1.png'),
  fs.readFileSync('./page2.png'),
  fs.readFileSync('./page3.png'),
];

await imagesToTiff(images , 'output.tiff');

// Get a Buffer (useful for uploading, streaming, etc.)
const tiffBuffer = await imagesToTiffBuffer(images);
fs.writeFileSync('./output.tiff', tiffBuffer);

That's it. No native binaries, no system dependencies. Works on AWS Lambda, Vercel, Docker, Windows — anywhere Node.js runs.

Use Cases
Multi-page TIFFs are common in:

Fax systems — fax files are almost always multi-page TIFFs
Document scanning — scanners output multi-page TIFFs
Medical imaging — DICOM-adjacent workflows often use TIFF
Legal/archival — PDF alternatives for scanned documents
Insurance and banking — document submission pipelines

Why Not Just Use PDF?
TIFF is lossless, supports higher bit depths (16-bit, 32-bit), and is required in many enterprise, government, and medical workflows where PDF is not accepted. If your system specification says TIFF, there's usually no substitute.

Links

npm: npmjs.com/package/multi-page-tiff
GitHub: github.com/yatindavra/multi-page-tiff

If you've been fighting with multi-page TIFF generation in Node.js, give it a try and let me know in the comments if you run into any issues.
check my work at https://yatindavra.vercel.app

DEV Community: Yatin Davra

I Built a Slot-Filling Onboarding Bot. It Leaked a Validation Error Into the Chat

The actual design problem

turnaround mode

The completion sentinel

The rule that actually fixes my bug

Where that leaves it

I Had a GBNF Grammar File and Nowhere to Point It. So I Added Support for It.

Why the existing schema types didn't fit

What I built

Then I tried to break my own validator

A real bug this surfaced

Where that leaves it

How I ended up contributing five ways to shape an LLM's output

How to Create Multi-Page TIFF Files in Node.js (Without ImageMagick)

`turnaround` mode