DEV Community

Cover image for Your model doesn't need to write the answer. It needs to point at it.
Smit Sojitra
Smit Sojitra

Posted on Originally published at smiit.in

Your model doesn't need to write the answer. It needs to point at it.

#ai #architecture #llm #machinelearning

You gave the LLM the answer. So why is it writing it out from scratch?

Send a model a support ticket and four labels, and ask which one fits. It writes back {"label": "billing"}, one token at a time. The word "billing" was in your prompt the whole time.

Extraction is the same. The invoice number is in the PDF, and the model types it back to you. Usually it types it right.

Writing is a loop, and a writer can write anything

An LLM reads your prompt in one pass, then writes the answer one token at a time, each token another trip through the model. That's also why output costs more: on Anthropic's price list, Sonnet 5.5 is $2 per million input tokens and $10 per million output.

For a short label the cost isn't the real problem. The problem is that a writer can write anything: "Billing", "billing_dispute", "Sure! The label is billing." So you add a parser, a retry, a schema. JSON mode helps, but it's "a format constraint, not a correctness guarantee," as one guide puts it.

A pointer works differently. It reads the input once and scores what's already there: each of your labels, or each possible span of the document. The answer can only be something from the input.

Two flows. Write: the input goes into a model that loops once per token and emits {

This isn't new. Pointer Networks (2015) picked elements of the input as output. SQuAD (2016) defined every answer as "a segment of text from the corresponding reading passage." BERT answered it by scoring each token as a possible start or end of the answer. Named entity recognition still returns start and end offsets today.

What ScaleDown's CDMs are

ScaleDown launched four small task models on October 13 and calls them Constrained Decoding Models. Their partner post with LangChain sums up the idea: "a CDM is trained for one task and only returns answers that task allows, like a label from your list or spans of the input."

Going by ScaleDown's docs:

  • Classify scores your text against each label's rubric and returns top_label plus scores that sum to 1.0. It points.
  • Extract returns "the exact text span extracted from the input," with a confidence and start/end offsets. It points.
  • Summarize is "a fluent rewrite - not a selection of lifted sentences." It writes.
  • Compress shrinks long context. The docs don't say whether it selects or rewrites.

So the idea holds for two of the four. One naming note: "constrained decoding" usually means masking a normal LLM's tokens so it can't break your schema (Outlines, XGrammar). Here it's a product name.

The real win is the receipt

Speed is nice. The bigger deal is that a pointed answer comes with proof. If Extract says the invoice number sits at characters 8 to 17, you slice your own string and check. A written answer with two digits swapped looks exactly like a correct one.

A made-up invoice line,

Where they'd sit in an app

Not instead of your main LLM. Around it. Small models do the routine jobs and the general model does what actually needs writing.

Architecture diagram. A user request goes to Classify, which routes it to the main LLM. Tool output goes through Compress before reaching the main LLM. The agent trace goes to a second Classify that checks for looping and can stop the run. Documents go to Extract, which writes spans with offsets into your database.

The loop check comes from LangChain's example: stop the run when Classify scores "looping" above 0.8. That only works if 0.8 means something on your data. ScaleDown doesn't publish calibration numbers, and Classify "always returns a winner - even if the model is uncertain," so add an other label and measure on a few hundred of your own examples.

Jev, OpenAI and Cloudflare do half of this

TypeSafe's Jev, OpenAI's Decisions API and Cloudflare's Clef all answer typed questions with probabilities. That's the Classify half. None of them document pointing at spans.

Four small platforms, one per API, each with three slots: pick a label, point at spans, summarize. ScaleDown CDMs fills all three, $0.05 per million input tokens. TypeSafe Jev ($0.042), OpenAI Decisions ($0.10, secondary source, beta) and Cloudflare Clef ($0.038 to $0.24) fill only pick a label; the other two slots are empty outlines for not documented. Output is free on all four

What pointing can't do

  • The answer has to be in the input. "How many items on this invoice?" needs adding up. A pointer has nothing to point at.
  • A valid label isn't a correct one. It can point at the wrong thing.
  • Summarize writes, and Compress doesn't say. Don't assume the "can't invent text" property covers them.
  • The big numbers are ScaleDown's. The tweet says 200x faster and 467x cheaper. ScaleDown's own benchmark docs say 8 to 19.5% more accurate and 2.2x to 13.9x faster. Nobody independent has checked either yet.
  • Your text leaves your servers. It's a hosted API.

Want the deeper dive? Check out the detailed blog with interactive examples here: smiit.in/writing/answer-is-in-the-input

Read more

Which of your LLM calls are secretly multiple choice? Tell me in the comments.

Top comments (1)

Collapse
 
deanlee profile image
Dean Lee •

The string slice is the underappreciated part here. With autoregressive extraction, the hidden cost is never the output token tariff alone, but the supervisory scaffolding you build to catch transposed digits or creative casing. Once you reduce extraction to start and end offsets, verification drops from fuzzy probabilistic checking to deterministic byte comparison.