DEV Community

Cover image for OpenRouter Returned 200, But My Next.js AI Stream Was Empty
nishchal singh
nishchal singh

Posted on Edited on

OpenRouter Returned 200, But My Next.js AI Stream Was Empty

My AI chat was working locally. In production, Vercel gave me a 504. After fixing that, OpenRouter returned 200 — and I still received nothing.

That was the point where I stopped looking for one broken line and started looking at the whole streaming path.

Original AI streaming architecture with custom SSE and NDJSON parsing in a Next.js application

The eventual problem was simpler than I expected:

I had too many layers handling the same stream.

Here's how I found it.

The setup

The application uses:

Next.js App Router
React
TypeScript
MongoDB / Mongoose
Clerk
TanStack Query
Cloudinary
OpenRouter
Enter fullscreen mode Exit fullscreen mode

The AI assistant is available at:

https://www.pssmumbai.org/ai
Enter fullscreen mode Exit fullscreen mode

It uses reviewed application knowledge to answer questions around PSS teachings, meditation and related topics.

The public chat endpoint is:

POST /api/ai/chat
Enter fullscreen mode Exit fullscreen mode

The model is:

openai/gpt-5-mini
Enter fullscreen mode Exit fullscreen mode

through OpenRouter.

There is quite a bit happening around the model request:

Authentication
Rate limiting
Quota
Conversation history
Knowledge retrieval
AI response
Conversation persistence
Usage tracking
Enter fullscreen mode Exit fullscreen mode

I wanted the response to stream to the browser instead of waiting for the entire answer.

That's where things had become unnecessarily complicated.

The streaming architecture I had built

The application already had an AI SDK implementation using:

streamText()
Enter fullscreen mode Exit fullscreen mode

and:

@openrouter/ai-sdk-provider
Enter fullscreen mode Exit fullscreen mode

But there was also an older custom streaming implementation.

The older path used:

src/lib/openai.ts
Enter fullscreen mode Exit fullscreen mode

That wrapper made its own request to OpenRouter and requested:

Accept: text/event-stream
Enter fullscreen mode Exit fullscreen mode

The API route then manually handled the provider response.

Debugging a production AI streaming failure from browser JSON error to Vercel timeout and zero stream events

It had to read chunks, parse SSE events, extract token deltas, detect completion and usage, accumulate the response, and convert everything into our own NDJSON format.

The browser then had another custom parser for that NDJSON stream.

So the flow looked roughly like this:

Browser
  ↓
fetch()
  ↓
Custom NDJSON parser
  ↓
Next.js API route
  ↓
Custom OpenRouter wrapper
  ↓
Custom SSE parser
  ↓
OpenRouter
  ↓
GPT-5 Mini
Enter fullscreen mode Exit fullscreen mode

There was also an AI SDK path sitting alongside it.

That was the first thing I should have questioned.

Multiple layers understood the same stream.

Then production failed

The first thing I saw in the browser was:

Unexpected token 'A', "An error o"... is not valid JSON
Enter fullscreen mode Exit fullscreen mode

At first, I thought I had a JSON problem.

Maybe the API was returning something unexpected.

Maybe the frontend parser was wrong.

Maybe the error response wasn't valid JSON.

So I opened the Network tab.

The request was:

POST /api/ai/chat
Enter fullscreen mode Exit fullscreen mode

and the response was:

504 Gateway Timeout
Enter fullscreen mode Exit fullscreen mode

That changed the investigation.

The JSON error wasn't the root problem.

The request itself wasn't completing.

Vercel was killing the function

The Vercel logs showed:

FUNCTION_INVOCATION_TIMEOUT
Enter fullscreen mode Exit fullscreen mode

with:

10.37s / 10s
Enter fullscreen mode Exit fullscreen mode

The function was being terminated before the AI request could finish.

I checked the repository for anything that might have configured this.

There was no:

vercel.json
Enter fullscreen mode Exit fullscreen mode

No route-level:

maxDuration
Enter fullscreen mode Exit fullscreen mode

and no runtime override that I had intentionally added.

So I checked the Vercel project settings.

Fluid Compute was disabled.

I enabled it and redeployed.

The function now had a much larger execution window.

The platform timeout was fixed.

But the request still wasn't working.

The timeout moved

After the deployment, the application started reaching its own timeout:

AI_TIMEOUT
Enter fullscreen mode Exit fullscreen mode

with:

The AI service took too long to respond. Please try again.
Enter fullscreen mode Exit fullscreen mode

The application-level timeout was around:

20 seconds
Enter fullscreen mode Exit fullscreen mode

That was useful information.

The problem had changed from:

Platform timeout
Enter fullscreen mode Exit fullscreen mode

to:

Application/provider streaming latency
Enter fullscreen mode Exit fullscreen mode

Increasing the function duration hadn't fixed the stream.

It had simply allowed the request to get further.

So I added more timing and streaming logs.

Then OpenRouter returned 200

This was the part that really confused me.

The request successfully passed the application stages:

auth_end
rate_limit_end
quota_reserve_end
conversation_lookup_end
retrieve_context_end
Enter fullscreen mode Exit fullscreen mode

Then:

openrouter_request_start
streaming=true
Enter fullscreen mode Exit fullscreen mode

OpenRouter responded:

openrouter_stream_headers
status=200
Enter fullscreen mode Exit fullscreen mode

So far, so good.

Except the stream counters said:

chunkCount=0
parsedEventCount=0
deltaCount=0
accumulatedChars=0
sawFinishReason=false
sawUsage=false
sawDone=false
Enter fullscreen mode Exit fullscreen mode

Eventually:

openrouter_stream_unexpected_eof
Enter fullscreen mode Exit fullscreen mode

The API itself eventually returned:

POST /api/ai/chat 200
Enter fullscreen mode Exit fullscreen mode

but the browser received:

The AI service disconnected before completing the response.
Enter fullscreen mode Exit fullscreen mode

So I had an HTTP 200 from OpenRouter, but zero usable streaming events reached my application.

That was the clue.

I couldn't reasonably say OpenRouter itself was broken.

The evidence pointed toward the streaming path I had built around it.

Simplified Next.js AI streaming architecture using AI SDK, OpenRouter and GPT-5 Mini

I stopped patching the parser

At this point, I could have kept modifying the custom SSE parser.

Instead, I wanted to answer one smaller question:

Can the AI SDK receive an OpenRouter stream correctly?

So I created an isolated test using:

AI SDK v6
+
@openrouter/ai-sdk-provider
+
OpenRouter
+
openai/gpt-5-mini
Enter fullscreen mode Exit fullscreen mode

The test successfully produced streamed deltas.

The result included:

success: true
model: openai/gpt-5-mini
deltasCount: 4
accumulatedChars: 21
finishReason: stop
usage present
providerRequestId present
Enter fullscreen mode Exit fullscreen mode

I wouldn't rely on the exact timing from that isolated test because the instrumentation around it wasn't reliable enough.

But I had enough evidence to move forward:

The AI SDK + OpenRouter combination could receive streamed output.

That changed the question.

Instead of asking:

Which line in my parser is broken?

I started asking:

Why am I maintaining this parser at all?

The actual problem was architectural

Looking back, I was handling too much myself.

The application had custom code for:

Provider streaming
SSE parsing
Transport conversion
Client stream parsing
Message accumulation
Cancellation
Completion handling
Enter fullscreen mode Exit fullscreen mode

None of those things are impossible to build.

The problem was that multiple layers had responsibility for the same stream.

The browser understood one streaming format.

The API route translated between formats.

The OpenRouter wrapper understood the provider stream.

The SSE parser interpreted provider events.

And the AI SDK had its own streaming abstraction.

More layers meant more places where data could be transformed, misunderstood or lost.

And when something failed, it wasn't immediately obvious which layer was responsible.

The fix: one authoritative streaming path

I stopped using the legacy custom streaming path for the public chat route.

The new flow is:

Browser
  ↓
POST /api/ai/chat
  ↓
Authentication
  ↓
Rate limiting
  ↓
Quota
  ↓
Conversation lookup
  ↓
Knowledge retrieval
  ↓
streamText()
  ↓
@openrouter/ai-sdk-provider
  ↓
OpenRouter
  ↓
GPT-5 Mini
  ↓
Stream back to browser
  ↓
Save completed response
  ↓
Persist usage
Enter fullscreen mode Exit fullscreen mode

Simplified Next.js AI streaming architecture using AI SDK, OpenRouter and GPT-5 Mini

The public route no longer calls:

getOpenAIClient().chat.completions.stream()
Enter fullscreen mode Exit fullscreen mode

and it no longer manually parses OpenRouter's raw SSE response.

The existing OpenAI-compatible wrapper wasn't removed completely because other non-streaming/admin functionality still uses it.

The change was specifically for the public streaming path.

The important decision was:

One implementation should be responsible for the model stream.

I didn't rewrite the frontend

I also didn't want to change everything at once.

The frontend already understood our NDJSON protocol:

{ "type": "delta", "text": "..." }
Enter fullscreen mode Exit fullscreen mode 
{ "type": "done", "..." }
Enter fullscreen mode Exit fullscreen mode 
{ "type": "error", "..." }
Enter fullscreen mode Exit fullscreen mode

I could have changed the provider, server transport, frontend parser and UI state management together.

That would have made the debugging harder.

Instead, I kept the existing browser contract and changed the provider-side streaming implementation first.

For the moment, the flow became:

AI SDK
  ↓
AI SDK stream
  ↓
Small compatibility layer
  ↓
Existing NDJSON client
Enter fullscreen mode Exit fullscreen mode

It's not the final architecture.

The longer-term plan is to move the frontend away from custom:

fetch() + NDJSON parsing
Enter fullscreen mode Exit fullscreen mode

and eventually use the AI SDK's React transport.

But that can be a separate change.

The first successful end-to-end test

After changing the public route, I tested the chat with:

Guide me through a 5-minute mindful meditation.
Enter fullscreen mode Exit fullscreen mode

The browser streamed the response successfully.

The logs showed:

ai_stream_selected
stream_path = ai_sdk_openrouter

stream_started

stream_first_text
firstDeltaMs = 10568
Enter fullscreen mode Exit fullscreen mode

The response was then persisted:

conversation_save:
~472 ms

usage persistence:
~166 ms
Enter fullscreen mode Exit fullscreen mode

And the request finished with:

stream_finished

POST /api/ai/chat 200 in 18584ms
Enter fullscreen mode Exit fullscreen mode

The old failure events were gone.

No:

openrouter_stream_unexpected_eof
Enter fullscreen mode Exit fullscreen mode

No:

openrouter_stream_error
Enter fullscreen mode Exit fullscreen mode

And the browser no longer showed the disconnect error.

The streaming path was working again.

But it was still slow

This is where the story gets more interesting.

The streaming problem was fixed.

The performance problem wasn't.

The successful request took approximately:

First text: ~10.568s
Total request: ~18.584s
Enter fullscreen mode Exit fullscreen mode

Ten seconds before the first useful text is still too long.

I don't want to hide that behind a nicer loading animation.

Now the question is:

Why is the first text taking around 10.5 seconds?

That's a different investigation.

I'm looking at:

Model reasoning configuration
OpenRouter/provider routing
Prompt size
Conversation history size
Retrieved context size
Duplicated instructions/context
Work happening before the provider call
AI SDK/provider configuration
Enter fullscreen mode Exit fullscreen mode

The model remains:

openai/gpt-5-mini
Enter fullscreen mode Exit fullscreen mode

I'm not changing the model just to get a better-looking number.

First I want to understand where the time is actually going.

Working and fast are different problems

This distinction became pretty important during this debugging session.

The first successful test answered:

Does the stream work?

Yes.

It didn't answer:

Is the stream fast enough?

Not yet.

Those are two different engineering problems.

So I'm now measuring:

Request start
Stream start
First text
Completion
Abort
Failure
Total latency
Enter fullscreen mode Exit fullscreen mode

I also want a request ID that follows the request through the important stages.

That should make the next performance investigation much easier than piecing unrelated log lines together.

What I'd change if I built it again

I wouldn't start by building a pile of streaming infrastructure.

I'd first decide which layer should own the stream.

For this application, the responsibilities are roughly:

Application
→ auth, quota, retrieval, persistence, business rules

AI SDK
→ model streaming and stream lifecycle

Provider
→ OpenRouter/model communication

UI
→ displaying and interacting with the response
Enter fullscreen mode Exit fullscreen mode

The mistake wasn't writing custom code.

The mistake was allowing several layers to own the same problem.

Once I removed that overlap, the system became much easier to reason about.

And that's probably the most useful thing I got from this debugging session:

Sometimes the fix isn't adding another layer.

It's deciding which layer doesn't need to be there.

Top comments (0)