Legacy Templates, Confidence Scoring, and Tracking What a Call Actually Cost

#ai #typescript #node #opensource

Not every integration gets to ask for clean JSON. Some systems only speak a rigid template, some answers are structurally fine but factually wrong, and some teams just need to know what all of this is actually costing them. Four more corners of shapecraft that don't show up in the quick-start.

1. Generating XML for a system that won't take anything else

Plenty of real integrations (SOAP endpoints, older enterprise systems, config formats) only accept XML, and only in a very specific shape. An XML template input lets you hand the model a real example with placeholder slots, and enforce that it only fills those slots in, not invents new tags:

const template = `<order>
  <customer></customer>
  <total></total>
</order>`;

const result = await generate(model, { xml: template, enforceLiterals: true }, prompt);

enforceLiterals is the part that matters here, it stops the model from quietly restructuring the document because it "improved" it.

2. Catching an answer that's valid JSON but just wrong

Structural validation only tells you the shape is right, not that the content makes sense. A confidence scorer runs after structural validation passes and can fail the attempt (triggering a retry) if the model's answer looks shaky:

const result = await generate(model, schema, prompt, {
  confidenceScorer: (value) => (value.summary.length < 10 ? 0.2 : 0.9),
  minConfidence: 0.5,
});

A semanticValidator does the same thing for outright content checks (does this diagnosis actually match the symptoms described), throwing to fail the attempt rather than scoring it.

3. Letting the model pick which operation to run

Sometimes you don't know ahead of time which of several typed operations a request needs, a support message might need a refund lookup, an account update, or just an FAQ answer. Skill-based generation registers each option and lets the model dispatch to the right one, arguments validated against that operation's own schema:

registry.register({
  name: "issue_refund",
  inputSchema: z.object({ orderId: z.string(), amount: z.number() }),
  handler: async ({ orderId, amount }) => processRefund(orderId, amount),
});

const call = await generateSkillCall(model, registry, userMessage);

Unlike provider-native tool calling, this works identically on every backend, including local models that have no tools API at all.

4. Knowing what a batch of calls actually cost

Once you're running this at any real volume, "it works" stops being the only question, "what did last night's run cost, and how slow was it" matters too. Every result carries metadata for exactly that:

const result = await generate(model, schema, prompt);
console.log(result.metadata);
// { provider: "openai", model: "gpt-4o-mini", latencyMs: 812, tokens: {...}, cost: 0.0004 }

Feed that into a createClient() logging middleware once, and every call in the app reports the same way without touching call sites.

Same core, more edges

XML for legacy systems, confidence scoring for content nobody double-checks by hand, skill dispatch for "which operation even applies here," cost tracking for the accountant asking questions later. None of it is a separate library bolted on, it's the same generate() retry-and-validate loop with a different knob turned.

Full docs for all of this, plus everything from the earlier posts, are up at aviasoletechnologies.github.io/shapecraft.

DEV Community