Streaming vs JSON: Trade-offs in AI-Powered Apps

This is a breakdown of a decision I made building LogicVisor, not a general "streaming vs JSON" guide. AI API responses aren't limited to text: some return images, audio, structured data. I only needed text, so streaming vs JSON became a real fork in the road. If I'd needed image responses too, this calculus changes entirely.

Two constraints were in tension from the start:

• The AI evaluation needed to be stored in the database.
• The same information needed to be available to the user immediately, at submission time.

Why you can't reliably stream JSON

JSON has to be parsed before it's usable, and you can't parse it in pieces as it arrives. It needs to be complete, start to finish, before JSON.parse() does anything with it. That forces a binary choice: stream, or don't. Streaming, in my case, meant raw text, because that's all the underlying APIs stream.

I started with streaming. Feedback was near-instant on the user end, and it felt great. Then I needed a more structured response, for both the user view and the database, so I switched the request over to ask for JSON instead of free text:

const groqParams: ChatCompletionCreateParamsNonStreaming = {
  messages: [{ role: "user", content: prompt.content }],
  model: groqModel,
  stream: false,
  response_format: {
    type: "json_object",
  },
};

const groqResponse = await groq.chat.completions.create(groqParams);
aiReviewText = groqResponse.choices[0].message.content ?? "";

stream: false plus response_format: json_object is the whole trick. The model still wraps its answer in a markdown code fence half the time regardless of what you ask for, so I still need to strip that off before parsing:

export function extractJSONFromMarkdown(markdownString: string) {
  let jsonString = markdownString.trim();

  if (jsonString.startsWith("```

json")) {
    jsonString = jsonString.substring(7);
  }
  if (jsonString.endsWith("

```")) {
    jsonString = jsonString.substring(0, jsonString.length - 3);
  }

  const parsedData: PromptResult = JSON.parse(jsonString.trim());
  return parsedData;
}

I got the data shape I wanted. What I lost was the user experience. Now the user was staring at a loading state for the full response time instead of watching text stream in.

What I actually learned

The streaming version was never faster in absolute terms. It just felt faster, because you're more patient when you can see progress happening. Streaming exploited that. JSON removed it.

Since I couldn't stream JSON (parsing requires a complete payload) and I couldn't get structured data out of raw streamed text without hacking something together after the fact, I stopped trying to force one approach to do both jobs. I fixed the UX problem directly instead: loading spinners, pulsers, rotating status text. Not real progress, but it reads as progress, and that's what the user actually responds to.

Two paths, two jobs

When I added anonymous, free-trial reviews, the storage requirements dropped. Anonymous users didn't need the same depth of data persisted as authenticated ones. So that path defaults to streaming, for the fastest perceived response, which is exactly what you want when you're trying to hook a new user during a free trial:

const groqStream = await groq.chat.completions.create({
  messages: [{ role: "user", content: prompt.content }],
  model: groqModel,
  stream: true,
});

for await (const chunk of groqStream) {
  const text = chunk.choices[0]?.delta?.content ?? "";
  fullText += text;

  controller.enqueue(
    encoder.encode(`data: ${JSON.stringify({ type: "content", content: text })}\n\n`),
  );
}

Same groq.chat.completions.create call as before, just stream: true instead of false, and no response_format. Each chunk gets pushed to the client over server-sent events as it arrives, and I accumulate fullText on the server side so I've still got the complete response to hand off for logging once the stream closes.

(The real implementation also retries on 503s with backoff and sends a "waking up" notice on cold starts. Left out here since it's not specific to the streaming vs JSON decision, just general resilience around a serverless AI provider.)

Authenticated users, who need full structured data persisted, get the non-streaming JSON path from the first section, feeding straight into a database write:

const result = extractJSONFromMarkdown(aiReviewText ?? "");

const review = await createAIReview({
  user_id: user.id,
  time_complexity: result?.time_complexity ?? "",
  overall_grade: result?.overall_grade ?? "",
  markdown_review: result?.markdown_review ?? "",
  submission_id: submission.id,
});

Two code paths, one architecture, each one earning its keep for a different constraint: speed for anonymous trial users, structure for authenticated ones.

The takeaway

Building AI-powered products (or honestly, any product) comes down to picking the trade-off that serves your actual goal, not the one that looks cleanest on paper. Streaming and JSON aren't competing solutions here, they're two tools solving two different problems in the same app.

---

Cover photo by Myles Bloomfield on Unsplash

Comments

[Marcus Kim]

The split between anonymous streaming reviews and authenticated JSON reviews is a practical product decision, especially since the latter feeds fields like overall_grade and time_complexity directly into durable storage. I also appreciated the admission that loading pulsers and rotating status text are only perceived progress; they improve patience, but they should not imply measurable completion if the backend can still stall on a cold start or retry. One useful extension would be to define an explicit cancellation and failure contract for both paths, because once streaming becomes the acquisition experience and JSON becomes the.

[Alex Shev]

Streaming feels better to users, but JSON is often easier to validate and recover from. I like treating this as a workflow choice: stream when the user needs progress and interruption, return structured JSON when the system needs durable state or downstream automation.