DEV Community

Cover image for Outlines: stop parsing broken JSON — constrain the tokens instead
M.T.Ramkrushna
M.T.Ramkrushna

Posted on

Outlines: stop parsing broken JSON — constrain the tokens instead

#aipythonllmdevops

Thursday AI Tool/Workflow. This week's afternoon tool: Outlines, the Apache-2.0 Python library from dottxt-ai for structured generation. You pass the type you want — int, Literal[...], a Pydantic model, a regex, or a Lark grammar — and Outlines steers the model so the completion matches that type.

If you've shipped an LLM feature that returns JSON, you know the other loop: ask for JSON, get a markdown fence, strip it, json.loads blows up, retry, then invent a "fix the JSON" prompt. Instructor and friends fix that after generation by validating and re-asking. Outlines (on local / steerable models) fixes it during generation by masking invalid tokens. Here's exactly how to use it.

What Outlines gives you

From the official docs (Outlines 1.3.3 on PyPI, Python ≥3.10,<3.14):

  • One call shape: model(prompt, output_type) — the second argument is the structure you want.
  • Five output-type families: basic Python types (int, bool, …), multiple choice (Literal / Enum / Choice), JSON schemas (Pydantic, dataclasses, TypedDict, JsonSchema), Regex, and CFG (Lark grammars).
  • Engines you already run: wrappers for Transformers, llama.cpp, vLLM (server + offline), Ollama, TGI, SGLang, MLX, plus API clients for OpenAI, Gemini, and others.
  • Generators: Generator(model, output_type) compiles the constraint once so repeated calls don't rebuild the logits processor every time (important for local models).

License is Apache-2.0. Docs live at dottxt-ai.github.io/outlines.

Retry-parse vs constrain-at-decode

Approach When it fails Cost
Prompt "return JSON" + json.loads fences, trailing commas, wrong keys silent bugs + retries
Validate + re-ask (e.g. Instructor) after a bad completion extra round-trips
Constrained decoding (Outlines on local/vLLM) tokens that break the schema are never sampled one pass; schema must be expressible

Use Outlines when you control the decoder (Transformers, llama.cpp, vLLM offline, etc.). On black-box APIs, Outlines still gives you the same model(prompt, type) API and forwards the schema to the provider's structured-output support where available — check the model feature matrix in the docs for which output types each backend supports.

1. Install and wrap a model 

pip install "outlines[transformers]"
# or: pip install outlines   then add the extra for your backend
Enter fullscreen mode Exit fullscreen mode

Local Transformers example from the getting-started guide:

import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "HuggingFaceTB/SmolLM2-135M-Instruct"
hf_model = AutoModelForCausalLM.from_pretrained(model_name)
hf_tokenizer = AutoTokenizer.from_pretrained(model_name)

model = outlines.from_transformers(hf_model, hf_tokenizer)
Enter fullscreen mode Exit fullscreen mode

Same idea for a server you already run (vLLM OpenAI-compatible, Ollama, TGI). Swap the factory:

import outlines
from openai import OpenAI
from ollama import Client

# vLLM server
vllm = outlines.from_vllm(
    OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY"),
    "microsoft/Phi-3-mini-4k-instruct",
)

# Ollama (model must already be pulled)
ollama_model = outlines.from_ollama(Client(), "tinyllama")
Enter fullscreen mode Exit fullscreen mode

2. Constrain the output type

Outlines mirrors Python type hints. The generator still returns a string; you cast it yourself.

from typing import Literal
from enum import Enum
from pydantic import BaseModel, Field
from outlines.types import Regex

# Basic type
n = model("How many minutes in one hour?", int)   # "60"

# Multiple choice
urgency = model(
    "Customer was charged twice this month. Urgency?",
    Literal["low", "medium", "high"],
)

class Priority(Enum):
    low = "low"
    medium = "medium"
    high = "high"

priority = model("Same ticket. Priority?", Priority)

# Regex (wrap the pattern)
ticket_id = model(
    "Extract the ticket id from: INC-00482 is open.",
    Regex(r"INC-[0-9]{5}"),
)

# Pydantic JSON schema
class Ticket(BaseModel):
    category: Literal["billing", "bug", "account", "feature_request"]
    urgency: Literal["low", "medium", "high"]
    summary: str = Field(description="One short sentence")

raw = model(
    "Classify: I was charged twice for my subscription this month.",
    Ticket,
    max_new_tokens=128,
)
ticket = Ticket.model_validate_json(raw)
print(ticket.category, ticket.urgency, ticket.summary)
Enter fullscreen mode Exit fullscreen mode

For dynamic choice lists (labels that change per request), use outlines.types.Choice([...]) instead of a hard-coded Literal.

3. Workflow: support-ticket router that never returns invalid JSON

Concrete loop you can drop into a FastAPI worker:

from outlines import Generator
from pydantic import BaseModel
from typing import Literal

class Route(BaseModel):
    team: Literal["billing", "tech", "account"]
    urgency: Literal["low", "medium", "high"]
    reason: str

# Compile once at process start
router = Generator(model, Route)

def route_ticket(text: str) -> Route:
    raw = router(
        f"Route this support ticket to a team.\n\nTicket:\n{text}",
        max_new_tokens=128,
    )
    return Route.model_validate_json(raw)

print(route_ticket("Refund still missing after 14 days."))
# Route(team='billing', urgency='high', reason='...')
Enter fullscreen mode Exit fullscreen mode

Why Generator matters: on steerable (local) models, building the logits processor from a schema is the expensive step. Compiling once at startup keeps request latency on the decode path.

Hypothetical production shape (same pattern as our earlier LiteLLM / Instructor posts):

  1. Local / vLLM model for classification + structured fields (cheap, constrained).
  2. Hosted frontier model only when urgency == "high" and you need a long reply.
  3. Log raw + parsed Route + model id so you can diff schema changes.

4. Gotchas before you ship

  • String in, typed out yourself. Outlines returns a string even for Pydantic / int. Always model_validate_json / int(...).
  • Not every backend supports every type. Regex and CFG need a steerable decoder. API models may only accept JSON schema / choice. Read the feature matrix for your from_* wrapper.
  • Schema design is the prompt. Overly wide str fields defeat the point. Prefer Literal, enums, bounded regexes, and short Field(description=...).
  • Compile once. Use Generator(model, output_type) for hot paths.
  • Pin the version. Outlines 1.3.3 pulls outlines_core==0.2.14; treat them as a pair in lockfiles.
  • Constraint ≠ correctness. The model can still pick the wrong team. Pair with a small labeled eval set (even 50 tickets) and measure accuracy separately from parse success (which should be ~100% under constrained decoding).

When to reach for Outlines vs the rest of the toolbox

  • Outlines — you own the decoder (or the provider's structured API) and invalid tokens are unacceptable.
  • Instructor / retry-validate — black-box chat APIs where you only get another round-trip.
  • DSPy (this morning's post) — the prompt itself should be optimized against a metric; Outlines can still be the structured leaf call inside a module.

References

Stop asking the model to "please return valid JSON." Tell the decoder which tokens are legal, then validate the string into your types.

Top comments (0)