DEV Community: Sathish Panthagani

Build a Document QA generator with Langchain and Streamlit

Sathish Panthagani — Wed, 22 Jan 2025 05:15:56 +0000

In this post, I will be explaining about with_structured_output api of langchain, which helps to output the response in the predefined output model.

At the end of this blog, we will be building a tool which generates a multiple choice questions along answers from the PDF uploaded. I find it is easy to host the solution on streamlit, it has one click deployment hosted on their community cloud and it is Free.

The demo app can be found here

Pre-requisites

Groq API key to access LLM models - you can get one from here
Streamlit account to host the application
Python experience

Tech Stack

Langchain - used to integrate LLM models
fitz - used to read the pdf file contents

Project Setup

Create a python virtual environment using venv with below command, read more about venv here

$ python -m venv .venv

activate virtual environment with below command

$ source <venv>/bin/activate

Create a file requirements.txt with below packages.

streamlit
streamlit-feedback
langchain
langchain-community
langchain-groq

Run pip install command to install the required packages.

pip install -r requirements.txt

Build Streamlit app

Create a file main.py and setup the basic streamlit app with below content

import streamlit as st
st.set_page_config(
    page_title="Document Questionnaire generator",
    page_icon="📚",
    layout="centered",
    initial_sidebar_state="auto",
)
st.title('Document Questionnaire generator')
st.caption("🚀 A Streamlit chatbot powered by Groq")
st.write('Generate multiple choice questionnaire from a pdf file uploaded')

Save file and run below command to run the streamlit app.

streamlit run main.py

The application automatically opens in browser window at http://localhost:8501/

initial streamlit app

Now add file upload control from streamlit to enable upload pdf feature.

file_bytes = st.file_uploader("Upload a PDF file", type=["pdf"])
Read the uploaded file and store the text content which we need to feed to LLM

import fitz

# Read the PDF
pages = []
pdf_document = fitz.open(stream=file_stream, filetype="pdf")
for page_num in range(len(pdf_document)):
    page = pdf_document.load_page(page_num)
    pages.append(page)

TEXT_CONTENT = " ".join(d.get_text() for d in pages)

Build LLM Model to generate question

Now we have the text content read from the pdf file, so we can go ahead with building LLM to generate Question answers from a prompt.
add below prompt to generate question and answers, we can also ask for number of questions generated.

# system prompt to generate questions
SYSTEM_PROMT =  """Generate {questions_count} multiple choice questions from the following text:
'''
{context}
'''
"""

SYSTEM_PROMT_FRMTD = SYSTEM_PROMT.format(context=TEXT_CONTENT, questions_count=NUM_QUESTIONS)

Invoke the LLM model from Groq,

from langchain_groq import ChatGroq

# Create a model
llm = ChatGroq(model="llama-3.1-70b-versatile", temperature=0, max_retries=2)

Since we would need a questions with answers we will be using with_structured_output api from langchain, this will output the results in a predefined structure so that it can be used for building an app.
Define the structure of the output model

# TypedDict
class MCQuestion(TypedDict):
    """Multiple choice question."""
    text: Annotated[str, ..., "The text of the question"]
    options: Annotated[list[str], ..., "The options for the question"]
    answer: Annotated[Optional[int], None, "the answer for the question from the options"]

class MultipleQuestion(TypedDict):
    """List of multiple choice questions."""
    questions: Annotated[list[MCQuestion], ..., "List of multiple choice questions"]

And then invoke LLM model with the text content from PDF.

# Cache the data so that it wont trigger the model to generate questions again
@st.cache_data
def call_llm(content: str) -> dict:
    """Call the model to generate questions."""
    # Generate questions
    structured_llm = llm.with_structured_output(MultipleQuestion)
    return structured_llm.invoke([SystemMessage(content=content)])


response = call_llm(SYSTEM_PROMT_FRMTD)

store questions to streamlit session state as streamlit run entire code for every change and we might lost the data.

st.session_state.questions = response.get("questions")

Once the questions are received from LLM output it can be presented in a multiple choice to the user and we can restrict user only after answering the question, once all questions are answered we can show a congratulations message.

for question in st.session_state.questions:
    answer_selected = st.radio(
        options=question.get("options"),
        label=question.get("text"),
        index=None)

    if not answer_selected:
        st.text('Please select an answer')
        st.stop()
    else:
        if question.get('options')[question.get('answer')] == answer_selected:
            st.text(f"✅ Correct! The answer is {answer_selected}")
        else:
            st.text('❌ Incorrect answer, please try again')
            st.stop()

st.balloons()
st.success('Congratulations! You have completed the questionnaire', icon="✔")

Once it tested working, you can push the code changes to GitHub.

In Streamlit, you have an option to deploy them by selecting the repository from GitHub and the starter file. Read more about Streamlit here

Thanks for reading and hope you enjoyed the article. You can find the code from Github which i have used in this article.

sathish39893 / streamlit-apps

streamlit apps built using generative ai llms

streamlit-apps

streamlit apps built using generative ai llms

View on GitHub

Building a React.dev RAG chatbot using Vercel AI SDK

Sathish Panthagani — Sun, 05 Jan 2025 19:12:42 +0000

Introduction

In this blog post, I'll share my journey of building a React.dev AI assistant which retrieves the info from react.dev website, and answers the prompt provided by the user. The main goal is to use the information available in react.dev website as knowledge base of the assistant.

Tech Stack

Next.js: A React-based framework for server-side rendering and routing.
ChatGPT: A popular generative AI LLM used for the assistant
Vercel AI SDK: The AI SDK is the TypeScript toolkit designed to help developers build AI-powered applications, follow more from their documentation here.
Supabase: A Postgres database to store vector embeddings with pgvector extension.
Tailwind CSS: A utility-first CSS framework that provides flexibility while keeping the UI clean and customisable.
Drizzle ORM: This library provides utilities to interact with DB and run vector similarity search against vector db.

Prerequisites:

This project requires below prerequisites to setup.

Supabase Database with Pgvector extension enabled - this will be used to store vector embeddings of the website content.
Open AI API key - if you dont have an OpenAI API key - get one from here

Please follow this link to enable pgvector extension for the database provisioned in supabase.

Project setup

This project consists of Next.js, Supabase, and Tailwind CSS as frontend UI, Vercel ai as backend chat integration with OpenAI API. It uses cheerio to scrap webpages and stores the embeddings in supabase pgvector database.

Creating NextJS Project

You can easily setup NextJS project by running following command or follow the instructions from NextJS from the (link)[https://nextjs.org/docs/app/getting-started/installation]

npx create-next-app@latest

Follow the screen instructions and it will create a NextJS project with basic setup.

Installing required libraries

npm install @ai-sdk/openai ai drizzle-orm drizzle-zod zod @langchain/community

Environment configuration

Create a .env file in the root directory of your project to manage variables used in the project.

Add the following variables, replacing the placeholders with the actual keys.

# .env

OPENAI_API_KEY=YOUR_OPENAI_API_KEY
DATABASE_URL=YOUR_SUPABASE_DATABASE_URL

RAG implementation

The assistant is based on Retrieval-Augmented Generation(RAG) which uses knowledge base based on the content from react.dev website. You can read more about RAG (here)[https://sdk.vercel.ai/docs/guides/rag-chatbot#what-is-rag]

Step 1: get the website content scrapped using cheerio plugin. The following code demonstrate the url as input and returns document content scrapped from the website.

import 'cheerio';
import { CheerioWebBaseLoader } from '@langchain/community/document_loaders/web/cheerio';
...
const pTagSelector = 'p';
const cheerioLoader = new CheerioWebBaseLoader(url, {
    selector: pTagSelector,
  });
const docs = await cheerioLoader.load();

Step 2: Split the website content into multiple chunks so that embeddings can be generated easily with shorter text.

import { RecursiveCharacterTextSplitter } from '@langchain/textsplitters';
...
const splitter = new RecursiveCharacterTextSplitter({
    chunkSize: 2000,
    chunkOverlap: 100,
  });
const allSplits = await splitter.splitDocuments(docs);
...

Step 3: generate the embeddings for each of the chunk and store them in vector db, so that it can be used to query from chat api.

const dbResult = await createResources(
    allSplits.map((doc) => ({
      content: doc.pageContent,
      source: doc.metadata.source,
    }))
  );

// actions/resources.ts
import { embed, embedMany } from 'ai';
import { openai } from '@ai-sdk/openai';

const embeddingModel = openai.embedding('text-embedding-ada-002');

export const generateEmbedding = async (value: string): Promise<number[]> => {
  const input = value.replaceAll('\\n', ' ');
  const { embedding } = await embed({
    model: embeddingModel,
    value: input,
  });
  return embedding;
};

export const createResources = async (values: NewResourceParams[]) => {
  try {
    for (const input of values) {
      const { content, source } = insertResourceSchema.parse(input);

      const embedding = await generateEmbedding(content);
      await db.insert(resources).values({
        content,
        source,
        embedding,
      });
    }

    return 'Resources successfully created and embedded.';
  } catch (error) {
    return error instanceof Error && error.message.length > 0
      ? error.message
      : 'Error, please try again.';
  }
};

Chat API implementation

once the knowledge base is built with react.dev website links, the document information get stored in supabase table. Using AI SDK, chat api can be implemented based their documentation available here.

following is the sample code to stream text from server:

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: openai('gpt-4o'),
    messages,
    system: `You are a helpful assistant. Check your knowledge base before answering any questions.
    Only respond to questions using information from tool calls.
    if no relevant information is found in the tool calls, respond, "Sorry, I don't know."`,
    tools: {
      getInformation: tool({
        description: `get information from your knowledge base to answer questions.`,
        parameters: z.object({
          question: z.string().describe('the users question'),
        }),
        execute: async ({ question }) => findRelevantContent(question),
      }),
    },
  });

  return result.toDataStreamResponse();
}

With the above Prompt, model will always looks for the knowledge base and triggers the tool getInformation which in turns executes function findRelevantContent. This function queries vector db and executes cosine similarity to find matches and returns the relevant context to LLM. LLM then use the knowledge base info and then generate response to the user query.

export const findRelevantContent = async (userQuery: string) => {
  const userQueryEmbedded = await generateEmbedding(userQuery);
  const similarity = sql<number>`1 - (${cosineDistance(
    resources.embedding,
    userQueryEmbedded,
  )})`;
  const similarGuides = await db
    .select({ name: resources.content, similarity })
    .from(resources)
    .where(gt(similarity, similarityThreshold))
    .orderBy(t => desc(t.similarity))
    .limit(4);
  return similarGuides;
};

Chat UI implementation

AI SDK has AI SDK UI which is framework-agnostic toolkit, streamlining the integration of advanced AI functionalities into your applications. It contains useChat hook which helps to integrate chat api (create earlier) with less effort.

// src/app/page.tsx

...
const { messages, input, handleInputChange, handleSubmit, error } = useChat({
    maxSteps: 10,
  });
...

Once UI part is implemented, it can be tested using any of the questions from react.dev website.

Conclusion

AI SDK has multiple features to integrate LLM directly to any of the react application without much hassle. It has utilities to for frontend and backend which makes it unique and easy to integrate.

In this guide, I have covered process of RAG implementation using AI SDK which will be helpful to create a knowledge base for any of the website.

A fully functional example of this project is included at the end of this article.

sathish39893 / react-docs-ai-app

react chat bot with RAG on react docs from react.dev website

React Docs Generative AI chatbot

This chatbot is a generative AI chatbot that can answer questions about React documentation using RAG (retrieval augmented generation) pipeline.

It is built using Next.js ai sdk and cheerio for web scraping and langchain to split the text into sentences. It uses drizzle ORM to store the embeddings of the scraped data and uses the embeddings to generate answers to the questions asked by the user.

Getting Started

First, run the development server:

npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev

Open http://localhost:3000 with your browser to see the result.

This project uses next/font to automatically optimize and load Geist, a new font family for Vercel.

Features

Generative AI Chatbot: The chatbot is built using Vercel's AI SDK and can answer questions about React documentation.
RAG (retrieval augmented generation) pipeline: The chatbot uses cheerio to…

View on GitHub