DEV Community

Cover image for I Built AI Streaming With Too Many Layers. Here’s What Broke
nishchal singh
nishchal singh

Posted on

I Built AI Streaming With Too Many Layers. Here’s What Broke

I was working on the AI assistant for an NGO application when I ran into a streaming problem that took me through a few different layers of the system.

The chat worked locally.

In production, it didn't.

Then I fixed a platform timeout and found another timeout. After that, I found something even stranger: OpenRouter was returning 200, but my application was receiving zero usable streaming events.

Eventually, I realized the bigger problem wasn't one broken request.

I had made the streaming architecture more complicated than it needed to be.

This is how I found it, what I changed, and what I'm still investigating.

The setup

The application uses Next.js App Router, React, TypeScript, MongoDB/Mongoose, Clerk, TanStack Query, Cloudinary and OpenRouter.

The AI assistant is available at:

https://www.pssmumbai.org/ai
Enter fullscreen mode Exit fullscreen mode

The public chat endpoint is:

POST /api/ai/chat
Enter fullscreen mode Exit fullscreen mode

The assistant uses reviewed application knowledge to answer questions around PSS teachings, meditation and related topics.

There is quite a bit happening before and after the model call:

Authentication
Rate limiting
Quota
Conversation history
Knowledge retrieval
AI response
Conversation persistence
Usage tracking
Enter fullscreen mode Exit fullscreen mode

For the model interaction, I use:

openai/gpt-5-mini
Enter fullscreen mode Exit fullscreen mode

through OpenRouter.

I wanted the response to stream to the browser rather than waiting for the complete answer.

That's where my architecture had become unnecessarily complicated.

The streaming setup I had built

The application already had an AI SDK implementation using:

streamText()
Enter fullscreen mode Exit fullscreen mode

and:

@openrouter/ai-sdk-provider
Enter fullscreen mode Exit fullscreen mode

But there was also an older custom streaming implementation.

The older path used an OpenAI-compatible wrapper in:

src/lib/openai.ts
Enter fullscreen mode Exit fullscreen mode

That wrapper made its own request to OpenRouter and requested:

Accept: text/event-stream
Enter fullscreen mode Exit fullscreen mode

The API route then manually handled the provider response.

It had to read chunks, parse SSE events, extract token deltas, detect completion and usage, accumulate the response, and eventually convert the result into our own NDJSON format.

The browser then had another custom parser to consume that NDJSON stream.

So the flow was roughly:

Browser
  ↓
fetch()
  ↓
Custom NDJSON parser
  ↓
Next.js API route
  ↓
Custom OpenRouter wrapper
  ↓
Custom SSE parser
  ↓
OpenRouter
  ↓
GPT-5 Mini
Enter fullscreen mode Exit fullscreen mode

Original AI streaming architecture with custom SSE and NDJSON parsing in a Next.js application

There was also an AI SDK path sitting alongside it.

That was the first thing I should have questioned.

The system had multiple layers that understood the same stream.

Then production failed

The first error I saw in the browser was:

Unexpected token 'A', "An error o"... is not valid JSON
Enter fullscreen mode Exit fullscreen mode

At first, I thought I had a JSON/API response problem.

Maybe the server was returning an unexpected response.

Maybe the frontend parser was wrong.

Maybe the error response itself wasn't valid JSON.

So I opened the Network tab.

The important part was:

POST /api/ai/chat
Enter fullscreen mode Exit fullscreen mode

returned:

504 Gateway Timeout
Enter fullscreen mode Exit fullscreen mode

The JSON error was only a symptom.

The actual request wasn't completing.

This was a good reminder that browser errors don't always tell you where the failure started. Sometimes you have to follow the request one layer deeper.

The Vercel timeout

The Vercel logs made the first problem much clearer:

FUNCTION_INVOCATION_TIMEOUT
Enter fullscreen mode Exit fullscreen mode

The execution was:

10.37s / 10s
Enter fullscreen mode Exit fullscreen mode

The function was being terminated before the AI request could complete.

I checked the repository for anything that might have configured this limit.

There was no:

vercel.json
Enter fullscreen mode Exit fullscreen mode

There was no route-level:

maxDuration
Enter fullscreen mode Exit fullscreen mode

and no runtime override that I had intentionally added.

So I checked the Vercel project settings.

Fluid Compute was disabled.

I enabled it and redeployed.

The function now had a much larger execution window.

That fixed the platform-level timeout.

But it also exposed another problem.

Fixing the timeout didn't fix the request

After the deployment, the application started reaching its own timeout instead.

The response was now a controlled application error:

AI_TIMEOUT
Enter fullscreen mode Exit fullscreen mode

with:

The AI service took too long to respond. Please try again.
Enter fullscreen mode Exit fullscreen mode

The application-level timeout was around:

20 seconds
Enter fullscreen mode Exit fullscreen mode

This was actually useful.

The first problem had been:

Platform timeout
Enter fullscreen mode Exit fullscreen mode

Now I was looking at:

Application/provider streaming latency
Enter fullscreen mode Exit fullscreen mode

Increasing the available function duration hadn't made the streaming implementation correct.

It had simply allowed the request to get further.

So I added more timing and streaming logs.

The strange part: OpenRouter returned 200

This was the most interesting trace.

The request successfully passed the application stages:

auth_end
rate_limit_end
quota_reserve_end
conversation_lookup_end
retrieve_context_end
Enter fullscreen mode Exit fullscreen mode

Then:

openrouter_request_start
streaming=true
Enter fullscreen mode Exit fullscreen mode

And OpenRouter responded:

openrouter_stream_headers
status=200
Enter fullscreen mode Exit fullscreen mode

That sounds good.

But immediately after that, the stream counters told a very different story:

chunkCount=0
parsedEventCount=0
deltaCount=0
accumulatedChars=0
sawFinishReason=false
sawUsage=false
sawDone=false
Enter fullscreen mode Exit fullscreen mode

Eventually:

openrouter_stream_unexpected_eof
Enter fullscreen mode Exit fullscreen mode

The API eventually returned:

POST /api/ai/chat 200
Enter fullscreen mode Exit fullscreen mode

but the browser received:

The AI service disconnected before completing the response.
Enter fullscreen mode Exit fullscreen mode

So I had an HTTP 200 from the provider, but no usable streamed content made it through my custom streaming path.

That was the clue I needed.

I couldn't reasonably conclude that OpenRouter itself was broken.

The evidence pointed me toward the path I had built around the provider.

Instead of patching the parser, I made a smaller test

Debugging a production AI streaming failure from browser JSON error to Vercel timeout and zero stream events

At this point, I could have kept modifying the custom SSE parser.

But I wanted a simpler answer first:

Can the AI SDK receive a stream from OpenRouter correctly?

So I created an isolated test using:

AI SDK v6
+
@openrouter/ai-sdk-provider
+
OpenRouter
+
openai/gpt-5-mini
Enter fullscreen mode Exit fullscreen mode

The test successfully produced streamed deltas.

The result included things like:

success: true
model: openai/gpt-5-mini
deltasCount: 4
accumulatedChars: 21
finishReason: stop
usage present
providerRequestId present
Enter fullscreen mode Exit fullscreen mode

I wouldn't rely on the exact timing from that isolated test because the instrumentation lifecycle around it wasn't reliable enough.

But the important result was much simpler:

The AI SDK + OpenRouter combination could successfully receive streamed output.

That changed how I looked at the problem.

I didn't necessarily need to find the exact line where my custom parser was failing.

I needed to ask whether I should have that parser in the first place.

The actual architectural problem

Looking back, I was solving too much myself.

The application had custom code handling:

Provider streaming
SSE parsing
Transport conversion
Client stream parsing
Message accumulation
Cancellation
Completion handling
Enter fullscreen mode Exit fullscreen mode

And there was already a mature abstraction available for the model streaming part.

The problem wasn't that custom streaming code is always wrong.

The problem was that I had multiple layers responsible for the same thing.

The browser understood one streaming protocol.

The API route translated between protocols.

The OpenRouter wrapper understood another streaming protocol.

The SSE parser interpreted provider events.

And the AI SDK had its own streaming abstraction.

That creates more places where a response can be transformed, misunderstood or dropped.

It also makes debugging harder because the error you see may be several layers away from where the problem started.

The change: one authoritative streaming path

I decided to stop using the legacy custom streaming path for the public chat route.

The new flow is much simpler:

Browser
  ↓
POST /api/ai/chat
  ↓
Authentication
  ↓
Rate limiting
  ↓
Quota
  ↓
Conversation lookup
  ↓
Knowledge retrieval
  ↓
streamText()
  ↓
@openrouter/ai-sdk-provider
  ↓
OpenRouter
  ↓
GPT-5 Mini
  ↓
Stream back to browser
  ↓
Save completed response
  ↓
Persist usage
Enter fullscreen mode Exit fullscreen mode

Simplified Next.js AI streaming architecture using AI SDK, OpenRouter and GPT-5 Mini

The public route no longer calls:

getOpenAIClient().chat.completions.stream()
Enter fullscreen mode Exit fullscreen mode

and it no longer manually parses OpenRouter's raw SSE response.

The existing OpenAI-compatible wrapper wasn't removed completely because other non-streaming/admin functionality still uses it.

The change was specifically about the public streaming path.

The important architectural decision was:

There should be one authoritative implementation responsible for the model stream.

I didn't change the frontend at the same time

There was another problem I wanted to avoid.

The frontend already understood our NDJSON protocol:

{ "type": "delta", "text": "..." }
Enter fullscreen mode Exit fullscreen mode 
{ "type": "done", "..." }
Enter fullscreen mode Exit fullscreen mode 
{ "type": "error", "..." }
Enter fullscreen mode Exit fullscreen mode

I could have changed the server transport, frontend parser and message handling in one large refactor.

I didn't.

For the first change, I kept the existing browser contract and replaced the provider-side streaming implementation.

So temporarily, the architecture became:

AI SDK
  ↓
AI SDK stream
  ↓
Small compatibility layer
  ↓
Existing NDJSON client
Enter fullscreen mode Exit fullscreen mode

This wasn't intended to be the final architecture.

It was a way to change one layer at a time and reduce the number of variables involved in the debugging.

The longer-term plan is to move the frontend away from custom fetch() + NDJSON parsing and eventually use the AI SDK's React transport.

But that can happen separately.

The first successful end-to-end test

After changing the public route, I tested the actual chat with:

Guide me through a 5-minute mindful meditation.
Enter fullscreen mode Exit fullscreen mode

The browser successfully streamed the response.

The application produced the expected answer, including the guided practice and knowledge sources.

The logs now showed:

ai_stream_selected
stream_path = ai_sdk_openrouter

stream_started

stream_first_text
firstDeltaMs = 10568
Enter fullscreen mode Exit fullscreen mode

The response was then persisted:

conversation_save:
~472 ms

usage persistence:
~166 ms
Enter fullscreen mode Exit fullscreen mode

And the request completed with:

stream_finished

POST /api/ai/chat 200 in 18584ms
Enter fullscreen mode Exit fullscreen mode

Just as important, the old streaming failure events were gone.

There was no:

openrouter_stream_unexpected_eof
Enter fullscreen mode Exit fullscreen mode

No:

openrouter_stream_error
Enter fullscreen mode Exit fullscreen mode

And the browser didn't show the disconnect message.

That gave me the first proper end-to-end confirmation that the new path was working.

But it's still too slow

This is where I want to be careful about the conclusion.

The streaming problem is fixed.

The performance problem isn't.

The successful request took approximately:

First text: ~10.568s
Total request: ~18.584s
Enter fullscreen mode Exit fullscreen mode

That's still too much waiting before the user sees the first useful text.

I don't want to hide that behind a nicer loading animation.

The next question is:

Why is the first text taking around 10.5 seconds?

That's now a performance investigation rather than a streaming correctness problem.

I'm looking at things such as:

Model reasoning configuration
OpenRouter/provider routing
Prompt size
Conversation history size
Retrieved context size
Duplicated instructions/context
Work happening before the provider call
AI SDK/provider configuration
Enter fullscreen mode Exit fullscreen mode

The model remains:

openai/gpt-5-mini
Enter fullscreen mode Exit fullscreen mode

I'm not planning to switch models just because another model might produce a nicer benchmark.

First I want to understand where the time is actually going.

What I learned from this

The biggest lesson wasn't that I should always use the AI SDK.

It's this:

If a mature library already handles a difficult infrastructure problem, think twice before rebuilding that responsibility yourself.

The application still needs to own plenty of things.

For example:

Authentication
Permissions
Quota
Rate limiting
Knowledge retrieval
Source filtering
Conversation persistence
Business rules
Observability
Enter fullscreen mode Exit fullscreen mode

Those are application concerns.

The model streaming abstraction doesn't need to know how the application's quota works or where its knowledge comes from.

Likewise, the application doesn't need to understand every detail of the provider's streaming protocol.

That separation makes the system easier to reason about.

Another lesson: don't change everything at once

One thing I'm glad I didn't do was rewrite the entire streaming system in one go.

The frontend already had a working NDJSON contract.

So I kept it temporarily.

That gave me a smaller change:

Old provider streaming
        ↓
New AI SDK provider streaming
        ↓
Existing browser contract
Enter fullscreen mode Exit fullscreen mode

Once that worked, I had a known-good baseline.

I can now replace the client transport separately.

This is especially useful when debugging systems with several moving parts.

If you change the provider integration, API response format, client parser and UI state management at the same time, a successful result doesn't tell you much about which change actually mattered.

Smaller changes give you better evidence.

Working and fast are two different problems

This distinction is probably the most useful thing I'll carry forward from this work.

The first successful test answered:

Does the stream work?

Yes.

It did not answer:

Is the stream fast enough?

Not yet.

Those are different engineering problems.

The next stage is to measure time to first text, completion time and the work happening before the provider request.

I also want better request-level observability around:

request ID
stream start
first text
completion
abort
failure
latency
Enter fullscreen mode Exit fullscreen mode

That should make the next performance investigation much less dependent on guesswork.

What's next

There are a few things I want to improve from here.

First, I want to understand the roughly 10-second first-text delay.

Then I'll look at the prompt, conversation history and retrieved context sizes, along with the model and provider configuration.

On the frontend, I eventually want to remove the custom NDJSON transport and move to the AI SDK React transport rather than maintaining our own client-side streaming state machine.

There is also UX work to do.

The current chat works, but the preparing/generating state still needs improvement. I'd like it to feel more natural while the application is waiting for the first response, without turning the interface into a collection of animations.

But that's separate from the architecture fix.

For now, the important part is that the stream itself is working again.

The part I'd change if I built it again

I wouldn't start with a pile of streaming infrastructure.

I'd first decide which layer should own the stream.

For this application, that means keeping the responsibilities roughly separated:

Application
→ auth, quota, retrieval, persistence, business rules

AI SDK
→ model streaming and stream lifecycle

Provider
→ OpenRouter/model communication

UI
→ displaying and interacting with the response
Enter fullscreen mode Exit fullscreen mode

The mistake wasn't writing custom code.

The mistake was allowing several layers to take responsibility for the same problem.

Once I removed that overlap, the system became much easier to understand.

And that's probably the part I'll remember from this debugging session:

Sometimes the useful architectural change isn't adding another layer.

It's deciding which layer doesn't need to be there.

Top comments (0)