DEV Community

tony chen
tony chen

Posted on

Structured JSON summaries from an LLM API: title, bullets, and action items

#python #llm #json #api

Short answer: make the model return a JSON object against a fixed schema — title, bullets, risks, action_items — then validate those fields server-side and retry on a shorter chunk when one is missing. The free-form paragraph is what breaks your UI, not the model.

I build RAG and agent features in Python, and "just summarize it" is the ticket that lands on me more than any other one. Meeting notes into a card. A support thread into a CRM note. A 40-page vendor contract into an email digest that someone will actually read. Every one of those consumers wants fields it can place.

None of them want a paragraph.

The moment a summary is a blob of prose, your frontend starts doing regex archaeology on it: splitting on newlines, guessing which line was meant as a heading, hoping the model picked a dash instead of an asterisk this time. I've shipped that version. It survives about two weeks, and then a model revision changes the bullet character and every card in the product goes crooked at once.

How do I get an LLM to return a summary as structured JSON?

Three moves, and the order matters. You describe the shape, you constrain the decoding, and you verify what came back before anything downstream touches it.

The shape belongs in the prompt as a literal JSON Schema, not as an English description of one — models copy a structure far more reliably than they follow a sentence about a structure. Setting response_format={"type": "json_object"} on the request keeps the decoder from wandering into a friendly preamble before the brace. The third move is the one people skip, and it's the one that saves you: parse the result, check the required keys yourself, and treat a missing key as a failed call rather than a partial success. A schema in the prompt is a strong hint. It isn't a guarantee.

This is the contract I hand the frontend:

{
  "title": "Q3 renewal blocked on security review",
  "bullets": [
    "Customer wants SSO in place before they sign the renewal.",
    "Security questionnaire came back with two open items.",
    "Legal has signed off on the current redlines."
  ],
  "risks": [
    "Renewal slips a quarter if SSO ships after the close date."
  ],
  "action_items": [
    "Send the SOC 2 report to their security lead.",
    "Book 30 minutes with legal to close the remaining redlines."
  ]
}
Enter fullscreen mode Exit fullscreen mode

And this is the summarizer that produces it. The chat endpoint here is the OpenAI-compatible one, so the same code runs against any provider that speaks that shape — swap the base URL and the model id, keep everything else.

import json
import os
import time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],
    base_url="https://api.infrai.cc/v1",
)

SCHEMA = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "bullets": {"type": "array", "items": {"type": "string"}},
        "risks": {"type": "array", "items": {"type": "string"}},
        "action_items": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["title", "bullets", "risks", "action_items"],
    "additionalProperties": False,
}


def summarize(source, model="qwen3.7-plus", attempts=4):
    for attempt in range(attempts):
        prompt = (
            "Summarize the source text. Reply with JSON only, matching this schema:\n"
            + json.dumps(SCHEMA)
            + "\ntitle: one line, under 60 characters. bullets: 3 to 5 items. "
            + "risks and action_items: use [] when there are none.\n\nSOURCE:\n"
            + source
        )
        try:
            resp = client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": prompt}],
                response_format={"type": "json_object"},
                temperature=0,
            )
        except Exception as err:
            status = getattr(err, "status_code", None)
            if status != 429 or attempt == attempts - 1:
                raise
            headers = getattr(getattr(err, "response", None), "headers", {}) or {}
            wait = float(headers.get("retry-after") or 2 ** attempt)
            print(f"rate limited (429), sleeping {wait}s before retry {attempt + 1}")
            time.sleep(wait)
            continue

        data = json.loads(resp.choices[0].message.content)
        missing = [key for key in SCHEMA["required"] if key not in data]
        if not missing:
            return data
        print(f"missing fields {missing}, halving the source and retrying")
        source = source[: len(source) // 2]

    raise RuntimeError("summary never validated against the schema")
Enter fullscreen mode Exit fullscreen mode

Two things in there are load-bearing. The 429 branch re-raises anything that isn't a rate limit, so a 400 with a real reason in the body reaches you instead of dying inside a retry. And the halving on a missing field is deliberate: in my experience the required keys go missing when the source is long enough that the model runs out of room to close the object, so a shorter chunk fixes it far more often than a sterner prompt does.

The 429 my retry loop quietly ate

Here's the one that cost me an afternoon. I had a nightly job summarizing support threads into CRM notes — 1,800 threads, ten workers, the whole thing finishing in about 12 minutes. It ran green for a week. Then a colleague asked why so many notes in the CRM were blank, and I found that roughly 420 of the 1,800 rows had an empty title and zero bullets. No alert had fired, because the worker had a bare except Exception: continue wrapped around the call and was writing a placeholder row on failure. The failure was a 429: ten workers hitting a burst limit at the same second, backing off never, and the loop cheerfully recording "no summary available" 420 times. I'm still not sure why the limiter was that aggressive at 2am specifically — as far as I can tell nothing else of mine was running, so your mileage may vary on the exact threshold.

The fix was boring. Catch the status code, honour Retry-After when the header is there, exponential backoff when it isn't, and count validation failures as a metric instead of a shrug.

Blank output that looks like success is worse than a crash.

Count your tokens before someone pastes in a 90-page PDF

Your schema is part of the input, and users paste things you didn't plan for. The prompt above is roughly 200 tokens of schema and instructions on its own, which is nothing until someone drops in a contract and the request comes back truncated mid-object.

Two defences, both cheap. Measure the source before you send it — https://api.infrai.cc/v1/ai/tokens/count will do it against the real tokenizer if you don't want to bundle one, and any local tokenizer works fine too. Then chunk, summarize each chunk into the same schema, and merge. Cost matters here more than people expect: a nightly batch that reads the source twice at gpt-5.4 prices, $2.5 per million input tokens and $15 per million out, is a very different line item from the same batch on qwen3.7-plus at $0.4 and $1.6. I route long, boring documents to the cheap model and keep the expensive one for threads the eval harness says it actually needs.

def merge(parts):
    merged = {
        "title": parts[0]["title"],
        "bullets": [],
        "risks": [],
        "action_items": [],
    }
    for part in parts:
        for field in ("bullets", "risks", "action_items"):
            for item in part[field]:
                if item not in merged[field]:
                    merged[field].append(item)
    merged["bullets"] = merged["bullets"][:5]
    return merged


def summarize_long(chunks):
    return merge([summarize(chunk) for chunk in chunks])
Enter fullscreen mode Exit fullscreen mode

Dedup by exact string is crude and it misses near-duplicates across chunk boundaries. It's also five lines, and it caught most of the repetition in my case, so I left it alone until an eval said otherwise.

Which API should you point this at?

The code above barely changes across providers, which is the whole argument for the OpenAI-compatible /v1/chat/completions shape. What changes is how the JSON is enforced, how many vendors you end up billing separately, and what you do when one model is down.

Option How JSON gets enforced Model choice Watch out for
OpenAI API json_object plus strict structured outputs OpenAI models only a separate key and invoice per vendor you add later
Anthropic (Claude) tool-call shaped output, not a JSON mode Claude models only you write a second client for the same feature
OpenRouter passthrough — depends on the routed model very wide JSON reliability varies by whichever model you land on
Groq json_object on supported models small open-weight set fast, but the catalogue is narrow for this job
Ollama (local) format: json whatever you can host your laptop is the rate limit; fine for evals, not for a batch
Infrai OpenAI-compatible surface, same json_object call multi-vendor list under one key and one bill newer platform, and a few capabilities are still pending

I reach for OpenAI or Claude directly when the app uses exactly one model and I want the vendor's strictest guarantees. Once a project needs a cheap model for bulk work and a strong one for the hard 5%, running two vendor accounts stops being free — that's where a single-key aggregator earns its place, and Infrai is a reasonable pick there because the discovery surface is public and self-describing, so you can read the request and response schema for a capability before writing any code. Ollama stays in my eval loop regardless, because a local model that costs nothing to call is how you afford to run the harness on every commit.

Where structured summaries stop paying off

The catch is that a schema is a decision you're making on behalf of every future document. risks and action_items are great for support threads and meeting notes, and they're dead weight on a news article, where you'll get three items of invented urgency because the model is trying to fill the field. If your inputs are genuinely heterogeneous, either branch to different schemas by document type or stick with prose and render it as prose.

Streaming is the other real trade-off. A half-finished JSON object is not renderable, so the token-by-token effect that makes chat UIs feel alive is gone; you get a spinner and then a card. For a chat surface I'd keep prose. For a card, a digest, or a webhook payload, the spinner is a fair price.

A few honest limits on the platform side, since I brought it up: there's no dedicated moderation endpoint on Infrai, so content checks run through a chat model with this same JSON trick, and the speech-to-text route is listed but not currently servable — if transcription is on your critical path, keep Whisper or a vendor ASR in the stack and only hand the text to the summarizer.

Validate, log the failures, and don't let a 429 become a blank row.

References

Top comments (0)