DEV Community

Olivier
Olivier

Posted on

7

Vercel AI SDK & LangChain streaming

If you got stuck like me for hours on how to use the new Vercel AI SDK with Langchain to stream, the trick is not to use the LangChainStream from Vercel's here https://sdk.vercel.ai/docs/api-reference/providers/langchain-stream

But to use the .stream with an outputparser here
https://sdk.vercel.ai/docs/guides/providers/langchain

Here is a custom LangChainStream function

langChainStreamCustom.ts

import { createStreamDataTransformer } from 'ai'

export const LangChainStreamCustom = (
  stream: any,
  { onCompletion }: { onCompletion: (completion: string) => Promise<void> }
): ReadableStream => {
  let completion = ''
  const transformStream = new TransformStream({
    async transform(chunk, controller) {
      completion += new TextDecoder('utf-8').decode(chunk)
      controller.enqueue(chunk)
    },
    async flush(controller) {
      await onCompletion(completion)
        .then(() => {
          controller.terminate()
        })
        .catch((e: any) => {
          console.error('Error', e)
          controller.terminate()
        })
    }
  })
  stream.pipeThrough(transformStream)
  return transformStream.readable.pipeThrough(createStreamDataTransformer())
}

Enter fullscreen mode Exit fullscreen mode

Then use it like all the other OpenAIStream AnthropicStream .. etc that Vercel SDK provides.

//....
response = llm.stream({}) // assuming this comes from a typical LangChain
const stream = LangChainStreamCustom(response, {
    onCompletion: async (completion: string) => {
      console.log('COMPLETE!', completion)
    }
})

return new StreamingTextResponse(stream)
Enter fullscreen mode Exit fullscreen mode

Enjoy.

Image of Datadog

The Future of AI, LLMs, and Observability on Google Cloud

Datadog sat down with Google’s Director of AI to discuss the current and future states of AI, ML, and LLMs on Google Cloud. Discover 7 key insights for technical leaders, covering everything from upskilling teams to observability best practices

Learn More

Top comments (3)

Collapse
 
nityasg profile image
Nitya Singh

Hey, I've been stuck on this problem for ages! Can you help me understand how can I use this for agents where we have agentexecutor.invoke() method.
Thanks!
Image description

Collapse
 
katsuma profile image
ryo katsuma

This article helped me a lot. Thanks!!!!!

Collapse
 
christophe_lombart_345911 profile image
Christophe Lombart

Not working for myself if I am using createStuffDocumentsChain. Any idea ?
Thanks

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more