{
"risk_level": "HIGH",
"requires_review": true,
"action_tier": "TIER_2"
}
Looks simple.
But a standard autoregressive LLM typically generates that response token by token.
{
↓
"risk_level"
↓
:
↓
"HIGH"
↓
,
↓
"requires_review"
↓
:
↓
true
↓
...
Every new token depends on the previous context.
But what if the schema is already known?
What if we already know that:
risk_level → HIGH | MEDIUM | LOW | NONE
requires_review → true | false
action_tier → TIER_1 | TIER_2 | TIER_3
Then generating every token of the JSON from scratch starts to look unnecessary.
This is where an interesting approach to structured LLM inference comes in.
🧠 The Core Idea
Instead of asking the LLM to generate the complete JSON sequence, we can treat the problem more like classification over constrained choices.
The model receives:
Context + JSON Schema
and the schema tells us what values are actually allowed.
For example:
risk_level
├── HIGH
├── MEDIUM
├── LOW
└── NONE
requires_review
├── true
└── false
action_tier
├── TIER_1
├── TIER_2
└── TIER_3
Now the model doesn't have to invent the structure.
It only needs to determine:
Which valid value is most likely for each field?
⚡ Step 1: Prefill the Context and Schema
The first step is to process the context and JSON schema through the Transformer.
Conceptually:
Context + Schema
↓
Transformer
↓
KV Cache
This is the prefill phase.
During this phase, the Transformer processes the input tokens and stores their keys and values in the KV cache.
The important part is that this information can be reused later.
We don't want to repeatedly process the same context if we don't have to.
🧩 Step 2: Reuse the KV Cache
Now consider the individual fields.
For example:
risk_level
requires_review
action_tier
For each field, we can reuse the cached context and process the field-specific suffix.
Conceptually:
KV Cache
│
┌─────────┼─────────┐
↓ ↓ ↓
risk_level requires_ action_tier
review
↓ ↓ ↓
Scores Scores Scores
↓ ↓ ↓
HIGH true TIER_2
Instead of repeatedly starting from the entire context, the model can reuse information that has already been computed.
🎯 Step 3: Look Only at Valid Candidates
A language model normally produces logits across its entire vocabulary.
That could mean tens of thousands of possible tokens.
But for:
risk_level
we only care about:
HIGH
MEDIUM
LOW
NONE
So why consider everything else?
We can focus on the candidate tokens relevant to that field.
For example:
Full vocabulary
↓
Filter candidates
↓
HIGH
MEDIUM
LOW
NONE
Then the candidate scores can be converted into probabilities.
Example:
HIGH → 0.9924
MEDIUM → 0.0068
LOW → 0.0006
NONE → 0.0002
The highest-probability valid candidate becomes the prediction:
HIGH
🔥 From Token Generation to Structured Prediction
This changes the way we think about the task.
Traditional LLM generation:
Prompt
↓
Token
↓
Token
↓
Token
↓
Token
↓
Valid JSON
Structured prediction:
Context + Schema
↓
KV Cache
↓
┌─────┼─────┐
↓ ↓ ↓
F1 F2 F3
↓ ↓ ↓
Score Score Score
↓ ↓ ↓
Value Value Value
└─────┼─────┘
↓
Valid JSON
The model is no longer responsible for inventing every bracket, comma, quote, key, and value.
The application already knows the structure.
The model focuses on the semantic decision.
🛡️ Another Big Advantage: Valid Structure
One common problem with LLM-generated JSON is that the model can produce something like:
{
"risk_level": "VERY_HIGH",
"requires_review": "maybe"
}
But our schema might only allow:
risk_level
→ HIGH | MEDIUM | LOW | NONE
requires_review
→ true | false
With constrained structured prediction, the candidate space itself can be restricted.
That means the system isn't simply asking:
"Please generate valid JSON."
It is enforcing the allowed structure around the model's prediction.
Conceptually:
LLM Vocabulary
↓
Schema Constraints
↓
Valid Candidates
↓
Probability Scores
↓
Selected Values
↓
Structured JSON
This makes the output much more predictable.
🏗️ The Architecture
A simplified architecture looks like this:
Input Document
│
▼
Context + Schema
│
▼
Transformer Decoder
│
▼
KV Cache
│
┌────────────┼────────────┐
▼ ▼ ▼
Field 1 Field 2 Field 3
│ │ │
▼ ▼ ▼
Candidates Candidates Candidates
│ │ │
▼ ▼ ▼
Logits Logits Logits
│ │ │
▼ ▼ ▼
Probabilities Probabilities Probabilities
│ │ │
└────────────┼────────────┘
▼
Structured JSON
The interesting part is the shared cached context.
🚀 Why KV Caching Matters
Imagine processing a large document.
The model might need to extract:
category
priority
risk_level
requires_review
action
confidence
If the same context is repeatedly processed from scratch, inference becomes unnecessarily expensive.
With KV caching:
Context
↓
Transformer
↓
KV Cache
↓
Reuse
┌─┼─┼─┼─┼─┐
↓ ↓ ↓ ↓ ↓ ↓
F1 F2 F3 F4 F5 F6
The expensive shared computation can be reused instead of recalculated for every field.
This is one of the reasons KV caching is such an important concept in LLM inference engineering.
📊 Where This Becomes Useful
This approach becomes especially interesting when your output schema is predictable.
Document Intelligence
{
"document_type": "INVOICE",
"priority": "HIGH",
"requires_review": true
}
Resume Analysis
{
"experience_level": "ENTRY",
"skill_match": "HIGH",
"recommendation": "SHORTLIST"
}
AI Agents
{
"tool": "SEARCH",
"priority": "HIGH",
"requires_confirmation": false
}
Classification Pipelines
{
"category": "TECHNICAL",
"sentiment": "POSITIVE",
"risk": "LOW"
}
In all of these cases, the possible values can be known before inference begins.
🧠 The Bigger AI Engineering Lesson
This idea made me think about something important.
When building LLM applications, we often ask:
How can I make the model generate better?
But another question is:
How can I make the model generate less?
Those are very different optimization strategies.
Instead of:
Generate everything
↓
Parse JSON
↓
Validate
↓
Fix errors
we can move toward:
Define schema
↓
Constrain candidates
↓
Reuse cached context
↓
Score valid choices
↓
Build structured output
The model handles the intelligence.
The system handles the structure.
⚙️ Traditional Generation vs Structured Inference
Traditional
Prompt
↓
LLM
↓
Token 1
↓
Token 2
↓
Token 3
↓
...
↓
JSON
↓
Validation
Structured
Context + Schema
↓
Prefill
↓
KV Cache
↓
Candidate Scoring
↓
Constrained Values
↓
Structured JSON
The second approach shifts more responsibility from free-form generation to the inference and application layer.
🔬 The Part I Find Most Interesting
The fascinating thing isn't just JSON.
It's the underlying idea:
A language model doesn't always need to behave like a text generator.
For some problems, it can behave more like a probabilistic decision engine operating inside a constrained space.
That opens up interesting possibilities around:
- constrained decoding
- structured generation
- KV-cache reuse
- inference optimization
- efficient batching
- model serving
- structured prediction
- AI agent orchestration
These are the kinds of details that become increasingly important when moving from simply using an LLM to actually engineering LLM systems.
💡 One Mental Model
The simplest way I'm thinking about it now is:
Traditional LLM
Understand
↓
Generate
↓
Generate
↓
Generate
↓
Validate
versus:
Structured LLM Inference
Understand
↓
Reuse Context
↓
Restrict Choices
↓
Score Candidates
↓
Select
↓
Build Valid Output
The model doesn't have to do everything.
Good AI engineering is often about deciding what the model should do — and what the system should handle instead.
🔥 Final Thought
LLM inference is much more than:
Prompt → Model → Response
Under the hood, concepts like:
Transformers → attention → KV cache → logits → softmax → constrained decoding → structured outputs
all contribute to how efficiently and reliably an AI system can operate.
The more I explore this layer, the more I realize that building AI applications isn't only about choosing a powerful model.
It's also about designing the inference process intelligently.
What other LLM inference optimization should I explore next? 👇
Top comments (0)