DEV Community: Dewy Knowledge Base

Building a RAG tool with Vercel's Generative UI components

Ryan Michael — Thu, 07 Mar 2024 19:52:11 +0000

Retrieval-Augmented Generation (RAG) blends the generative abilities of LLMs with the retrieval of information from diverse knowledge bases (KBs). However, traditional implementations of RAG have often relied on pre-scripting the logic for selecting and utilizing these KBs.

This conventional method, while effective, places limits on the flexibility and adaptability, and performance of the applications.

But consider an alternative: applications that empower the LLM itself to determine which actions are necessary to generate a response. This approach not only harnesses the LLM's generative capabilities but also its ability to make contextual decisions on-the-fly, thereby opening up a more dynamic and responsive way to handle knowledge search tasks. As a result, you can build targeted KB's and use the power of LLMs to write application-specific queries that retrieve better, more relevant results.

To demonstrate this approach, we'll build a simple chatbot backed by a custom tool. The chatbot will have access to a KB containing product information - any time you ask a question the chatbot determines could be answered using this tool it will query the KB and show both the in-progress query and retrieved results as custom UI elements before responding with the retrieved products as context.

We'll use Vercel's Generative UI library to render the chat window and Dewy to implement the information storage and retrieval used by our tool and Dewy as the knowledge store backing the tool.

Why Dewy and Vercel?

Vercel's Generative UI library) is designed to simplify the creation of custom LLM "tools" and to render rich, interactive UI elements that are specific to each tool the LLM chooses to use. What this means in practice is that your UI can dynamically adapt to the specific tools chosen by the LLM, displaying tool-specific information and progress indicators to the user.

Dewy is an OSS knowledge base with the simplicity and ease-of-use of a document store: insert your documents and Dewy takes care of preparing them for semantic search. Dewy's flexibility and ease of use allow developers to focus on building amazing user experiences rather than complex data-processing pipelines.

Prerequisites

Before diving into the tutorial, ensure you have the following prerequisites covered:

Basic knowledge of Typescript and React
A NextJS environment setup on your local machine
A copy of Dewy running on your local machine (see Dewy's installation instructions if you need help here).
Access to the OpenAI API platform.

Set up your project

Initialize a new NextJS project: Create a NextJS app by running the following command in your terminal:
```
pnpm dlx create-next-app@canary rag-tool
```
Navigate into your new project directory:
```
cd rag-tool
```
Install required packages: Next, we'll install ai, Vercel's AI library, openai, OpenAI's official JavaScript SDK compatible with the Vercel Edge Runtime, and dewy-ts, the Dewy client library. Zod will be used to describe the input types for our custom tool.
```
pnpm install ai openai zod dewy-ts
```
Prepare your environment: Configure your OpenAI key and Dewy endpoint.
```
// .env.local
OPENAI_API_KEY=xxxxxxxxx
DEWY_ENDPOINT=localhost:8000
```

Create a custom tool

First, create the custom tool the LLM will use to answer product-related questions. This tool describes its purpose and parameters so the LLM knows when and how to use it, and defines the implementation and UI logic to complete the tool invocation.

In this case, the tool will "get information about products", given a search query and a result count. When the LLM determines this tool should be used, it will generate the query string and choose how many results are appropriate, then execute the render function defined below.

On execution, the tool searches Dewy for relevant products, then calls the LLM again to pick up where it left off. Since this process can take awhile, the render function yield's components indicating that the tool is being used and its progress. These components allow the UI to reflect the specific information relevant to this tool's outcomes.

// app/productSearch.tsx

import { z } from 'zod';
import { Dewy } from 'dewy-ts';
import { OpenAIStream } from 'ai'
import { Tokens } from 'ai/react';

import AssistantMessage from './AssistantMessage';
import ResultCard from './ResultCard';
import SearchCard from './SearchCard';

// Create a Dewy client.
const dewy = new Dewy({
    BASE: process.env.DEWY_ENDPOINT
})

// Implement the tool's logic.
// In this case, we search for the given query
// and return the `count` most similar chunks in the KB.
// The returned chunks are used in the `render` method below.
async function searchProducts(query: string, count: number) {
    const context = await dewy.kb.retrieveChunks({
        collection: "product_info",
        query: query,
        n: count,
    })
    return context.text_results.map(c => c.text)
}

// Define the behavior of the product search tool.
export default function productSearch(aiState, openai) {
    return {
        // A description of the tool.
        // This used by the LLM to decide when to use the tool
        description: 'Get information about products',

        // Parameters control how the tool behaves.
        // These values will be picked by the LLM,
        // so be sure to clearly explain what they're
        // used for.
        parameters: z.object({
            query: z.string().describe(```


                A description of a product 
                or what the product can be used for.


            ```),
            count: z.number().describe(```


                The number of products to return.


            ```),
        }).required(),

        // Configure the tool's behavior.
        // This function will be called after the LLM has
        // chosen to use the tool and generated values for
        // the parameters we configured above.
        render: async function* ({ query, count }) {
            // Let the user know we're looking for 
            // products related to their message
            yield <SearchCard query={query} count={count} />

            // Search for products related to the user's question
            const products = await searchProducts(query, count)

            // Update the message history 
            // with the results we found
            aiState.update([
                ...aiState.get(),
                {
                    role: "function",
                    name: "product_search",
                    content: JSON.stringify(products),
                }
            ]);

            // Now reply to the user.
            // The products we retrieved are part of the state 
            // provided as the messages parameter
            const resp = await openai.chat.completions.create({
                model: 'gpt-4-0125-preview',
                messages: aiState.get(),
                stream: true,
            })

            // Stream the results back as they're generated
            const stream = OpenAIStream(resp, {
                onFinal: (completion) => {
                    // Update the conversation history 
                    // once the full response is received
                    aiState.done([
                        ...aiState.get(),
                        {
                            role: "assistant",
                            content: completion,
                        }
                    ])
                }
            });

            // Display the response alongside 
            // the products provided to the model.
            return <div className="flex flex-col gap-2">
                <ResultCard query={query} results={context} />
                <AssistantMessage>
                    <Tokens stream={stream} />
                </AssistantMessage>
            </div>
        }
    }
}

Setup the server-side message handler

Configure a message handling action to receive new messages from the client. This handler configures the initial state of the chatbot and defines the server's behavior when a message is received from the user.

// app/action.tsx

import { OpenAI } from 'openai';
import { createAI, createStreamableUI, getMutableAIState, render } from 'ai/rsc';

import productSearch from './productSearch';
import AssistantMessage from './AssistantMessage';

// Configure the LLM, in this case OpenAI
const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

async function submitUserMessage(userInput: string) {
  'use server';

  // The AI state. This contains our message history,
  // and will be provided to the LLM each time a 
  // chat completion is generated.
  const aiState = getMutableAIState<typeof AI>();
  aiState.update([
    ...aiState.get(),
    {
      role: 'user',
      content: userInput,
    },
  ]);

  // Helper function for building streamable UI components.
  const ui = render({
    model: 'gpt-4-0125-preview',
    provider: openai,
    messages: [
      { role: 'system', content: `
        You are a product recommendation engine. 
        Respond only with information about products 
        retrieved using the "product_search" function role.
        `},
      { role: 'user', content: userInput }
    ],

    // This determines how generated text 
    // (as opposed to function calls) will
    // be rendered.
    text: ({ content, done }) => {
      if (done) {
        aiState.done([
          ...aiState.get(),
          {
            role: "assistant",
            content
          }
        ]);
      }
      return <AssistantMessage>{content}</AssistantMessage>
    },

    // Configure the tools available to the LLM
    tools: {
      product_search: productSearch(aiState, openai),
    }
  })

  return {
    id: Date.now(),
    display: ui
  };
}

// Define the initial state of the AI. It can be any JSON object.
const initialAIState: {
  role: 'user' | 'assistant' | 'system' | 'function';
  content: string;
  id?: string;
  name?: string;
}[] = [];

// The initial UI state that the client will keep track of, which contains the message IDs and their UI nodes.
const initialUIState: {
  id: number;
  display: React.ReactNode;
}[] = [{id: 0, display: <AssistantMessage>Hi! How can I help you today?</AssistantMessage>}];

// AI is a provider you wrap your application with so you can access AI and UI state in your components.
export const AI = createAI({
  actions: {
    submitUserMessage
  },
  // Each state can be any shape of object, but for chat applications
  // it makes sense to have an array of messages. Or you may prefer something like { id: number, messages: Message[] }
  initialUIState,
  initialAIState
});

Setup the chat UI on the client

Finally, set up the chat UI on the client.
Map over the UI state's messages and insert their display property. Configure a form to update the AI state, then call the submitUserMessage server side handler.

// app/page.tsx
'use client'

import { useState } from 'react';
import { useUIState, useActions } from 'ai/rsc';
import type { AI } from './action';
import { Input } from '@/components/ui/input';
import { Button } from '@/components/ui/button';

import UserMessage from './UserMessage';
import { PlaneIcon } from './icons';

export default function Page() {
  const [inputValue, setInputValue] = useState('');
  const [messages, setMessages] = useUIState<typeof AI>();
  const { submitUserMessage } = useActions<typeof AI>();

  return (
    <>
      <div className="flex-1 overflow-auto px-4 mt-4">
        <div className="grid gap-4 md:gap-8">
          {
            messages.map((message) => (
              <div key={message.id}>
                {message.display}
              </div>
            ))
          }

        </div>
      </div>

      <form onSubmit={async (e) => {
        e.preventDefault();

        // Add user message to UI state
        setMessages((currentMessages) => [
          ...currentMessages,
          {
            id: Date.now(),
            display: <UserMessage>{inputValue}</UserMessage>,
          },
        ]);

        // Submit and get response message
        const responseMessage = await submitUserMessage(inputValue);
        setMessages((currentMessages) => [
          ...currentMessages,
          responseMessage,
        ]);

        setInputValue('');
      }}>
        <div className="border-t-2">
          <div className="flex items-center h-14 px-4">
            <Input
              className="rounded-full flex-1 min-w-0 bg-gray-200 dark:bg-gray-800"
              placeholder="Type a message..."
              value={inputValue}
              type="text"
              onChange={(event) => {
                setInputValue(event.target.value)
              }}
            />
            <Button className="ml-2 h-8 w-8 rounded-full bg-gray-200 dark:bg-gray-800" size="icon">
              <PlaneIcon className="h-4 w-4" />
              <span className="sr-only">Send message</span>
            </Button>
          </div>
        </div>
      </form>
    </>
  )
}

To keep this post from going any longer than it is already, several basic UI components have been omitted, but you can checkout the full implementation in the examples directory of the Dewy repo.

Conclusion

In conclusion, this blog post has guided you through the process of building a Retrieval-Augmented Generation (RAG) tool using Vercel's Generative UI components. This approach moves away from the traditional methods of pre-scripting logic for selecting and utilizing knowledge bases. By empowering the LLM to determine which knowledge bases are necessary and how best to use them, we've showcased a more dynamic, efficient, and user-centric way of handling information retrieval tasks.

By utilizing Dewy as our knowledge base, we've emphasized the importance of a flexible, easy-to-use platform for storing, organizing, and retrieving information. This synergy between Dewy's streamlined data management and Vercel's dynamic UI rendering paves the way for developers to create more intelligent, responsive, and user-friendly applications.

If this tutorial has been helpful and you'd like to help others learn about Dewy, please consider starring our GitHub repo!

Building a Question-Answering CLI with Dewy and LangChain

Ben Chambers — Mon, 04 Mar 2024 18:10:42 +0000

In this tutorial, we're focusing on how to build a question-answering CLI tool using Dewy and LangChain. Dewy is an open-source knowledge base that helps developers organize and retrieve information efficiently. LangChain is a framework that simplifies the integration of large language models (LLMs) into applications. By combining Dewy's capabilities for managing knowledge with LangChain's LLM integration, you can create tools that answer complex queries with precise and relevant information.

The use of a knowledge base to augment the capabilities of an LLM is referred to as retrieval augmented generation or RAG. This guide walks you through setting up a simple command-line RAG application. It covers everything from setting up your environment andi loading documents into Dewy to using an LLM through LangChain to answer questions based on the retrieved results. It's designed for engineers looking to enhance their projects with advanced question-answering functionalities.

Why Dewy and LangChain?

Dewy is an OSS knowledge base designed to streamline the way developers store, organize, and retrieve information. Its flexibility and ease of use make it an excellent choice for developers aiming to build knowledge-driven applications.

LangChain, on the other hand, is a powerful framework that enables developers to integrate LLMs into their applications seamlessly. By combining Dewy's structured knowledge management with LangChain.js's LLM capabilities, developers can create sophisticated question-answering systems that can understand and process complex queries, offering precise and contextually relevant answers.

The Goal

Our aim is to build a simple yet powerful question-answering CLI script. This script will allow users to load documents into the Dewy knowledge base and then use an LLM, through LangChain, to answer questions based on the information stored in Dewy. This tutorial will guide you through the process, from setting up your environment to implementing the CLI script.

You'll learn how to use LangChain to build a simple question-answering application, and how to integrate Dewy as a source of knowledge, allowing your application to answer questions based on specific documents you provide it.

Prerequisites

Before diving into the tutorial, ensure you have the following prerequisites covered:

Basic knowledge of Python programming
Familiarity with CLI tools development
A copy of Dewy running on your local machine (see Dewy's installation instructions if you need help here).

Step 1: Set Up Your Project

The final code for this example is available in the Dewy repo if you'd like to jump ahead.

First, create a directory for the CLI project and change into the directory

mkdir dewy_qa
cd dewy_qa

With the directory set up, you can create and initialize a project using Poetry:

poetry init

When it asks about defining your main dependencies interactively you can choose yes and enter the following:

langchain-core which we'll use for the orchestration
langchain-openai which we'll use for the OpenAI LLM interface
click which we'll use for the CLI
dewy-langchain which provides the LangChain retriver querying Dewy

Now you're ready to create the CLI application.

We're using click, which lets us create a CLI using decorators on methods.
To start, we'll create a "group" for the two commands we're going to implement -- one for adding a document and one for asking a question.

``python title="CLI entry point" @click.group() @click.option("--collection", default="main") @click.option("--base_url", default="http://localhost:8000") @click.pass_context def cli(ctx, collection, base_url): # ensure that ctx.obj exists and is a dict (in casecli()is called # by means other than theif` block below)
ctx.ensure_object(dict)

ctx.obj["base_url"] = base_url
ctx.obj["collection"] = collection

Commands will go here

if name == "main":
cli()




In addition to creating a group, this does the following:

- Accepts a `collection` argument indicating which Dewy collection to operate on.
- Accepts a `base_url` argument (with a default) indicating which Dewy service to connect to.
- Stores both of those options on the context.
- Executes the `cli` group when invoked.

Adding these options to the root allows them to be passed before the specific command, and makes it clear they apply to all (or most) of the commands in the CLI application we're building.

Now you can run your script with `poetry run python -m dewy_qa`.

## Step 2: Implement document loading

Load documents by setting up the Dewy client. The following code adds an `add_file` command which accepts a single positional `url_or_file`. If that corresponds to a valid file path, it uploads the file to Dewy. Otherwise, it creates the document from the given URL and Dewy will fetch it. This logic could be improved (eg., `file://` URLs should be uploaded) but it demonstrates several key abilities:

1. You can create a document from a URL, and Dewy will download and ingest the file.
2. You can create a document without associated content, and then upload content which Dewy will ingest.



```python title="Add File Command"
@cli.command()
@click.pass_context
@click.argument("url_or_file")
def add_file(ctx, url_or_file):
    from dewy_client.api.kb import add_document, upload_document_content
    from dewy_client.models import AddDocumentRequest, BodyUploadDocumentContent
    from dewy_client.types import File

    client = Client(ctx.obj["base_url"])
    if os.path.isfile(url_or_file):
        document = add_document.sync(
            client=client,
            body=AddDocumentRequest(
                collection=ctx.obj["collection"],
            ),
        )
        print(f"Added document {document.id}. Uploading content.")

        with open(url_or_file, "rb") as file:
            payload = file.read()
            upload_document_content.sync(
                document.id,
                client=client,
                body=BodyUploadDocumentContent(
                    content=File(
                        payload=payload,
                        file_name=os.path.basename(url_or_file),
                    ),
                ),
            )
        print(f"Uploaded content for document {document.id}.")

    else:
        document = add_document.sync(
            client=client,
            body=AddDocumentRequest(collection=ctx.obj["collection"], url=url_or_file),
        )
        print(f"Added document {document.id} from URL '{url_or_file}'")

At this point, you should be able to load a document from a URL or local PDF using the command poetry run python -m dewy_qa <url_or_file>.
For example, you could use https://arxiv.org/pdf/2009.08553.pdf to load a PDF from Arxiv.

You may ask -- "what happens if I upload content to a document that is already ingested?" Conveniently, Dewy will treat this as a new version of the document and re-ingest it!

Step 3: Implement question-answering

With the ability to load documents into Dewy, it's time to integrate LangChain to invoke LLMs for answering questions. This step involves setting up LangChain to query the Dewy knowledge base and process the results using an LLM to generate answers.

We're going to introduce a query command which accepts a file containing the question (or reads it from stdin).
We'll build this up in several steps.

Create DewyRetriever

First, we'll create the query command and create a DewyRetriever for our collection.
This is an adapter that let's LangChain know how to retrieve documents from Dewy.

```python title="Create DewyRetriever"
from dewy_langchain import DewyRetriever

retriever = DewyRetriever.for_collection(
collection=ctx.obj["collection"], base_url=ctx.obj["base_url"]
)




### Create a PromptTemplate


This is a string template that tells LangChain how to create the prompt for the LLM. In this case, the LLM is instructed to answer the question, but only using the information it's provided. This reduces the model's tendency to "hallucinate", or make up an answer that's plausible but wrong.  The values of `context` and `question` will be configured when we assemble the "chain".



```python title="Prompt Template"
from langchain_core.prompts import ChatPromptTemplate
prompt = ChatPromptTemplate.from_template(
    """Answer the question based only on the following context:
{context}

Question: {question}
"""
)

Create the Chain

LangChain works by building up "chains" of behavior that control how to query the LLM and other data sources. This example uses LCEL, which provides a more flexible programming experience than some of LangChain's original interfaces.

Use a RunnableSequence to create an LCEL chain. This chain describes how to generate the context and question values: the context is generated using the retriever created earlier, and the question is generated by passing through the step's input. The results Dewy retrieves are formatted as a string by piping them to the formatDocumentsAsString function.

This chain does the following:

It retrieves documents using the DewyRetriever and assigns them to context and assigns the chain's input value to question.
It formats the prompt string using the context and question variables.
It passes the formatted prompt to the LLM to generate a response.
It formats the LLM's response as a string.

```python title="Create the Chain"
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI

model = ChatOpenAI()

chain = (
{"context": retriever, "question": RunnablePassthrough()}
| prompt
| model
| StrOutputParser()
)




### Invoke the Chain and Print Results

Now that the chain has been constructed, execute it and output the results to the console. As you'll see, `question` is an input argument provided by the caller of the function.

Executing the chain using `chain.streamLog()` allows you to see each response chunk as it's returned from the LLM. The stream handler loop is sort of ugly, but it's just filtering to appropriate stream results and writing them to `STDOUT` (using `console.log` it would have added newlines after each chunk).



```python title="Invoke the Chain"
query_str = query.read()
click.echo(f"Invoking chain for:\n{query_str}")
result = chain.invoke(query_str)
click.echo(f"\n\nAnswer:\n{result}")

Putting it all together

```python title="Putting it all together"
@cli.command()
@click.pass_context
@click.argument("query", type=click.File("r"), default=sys.stdin)
def query(ctx, query):
from dewy_langchain import DewyRetriever
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI

retriever = DewyRetriever.for_collection(
    collection=ctx.obj["collection"], base_url=ctx.obj["base_url"]
)

prompt = ChatPromptTemplate.from_template(
    """Answer the question based only on the following context:
{context}

Question: {question}
"""
)

model = ChatOpenAI()

chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | model
    | StrOutputParser()
)

query_str = query.read()
click.echo(f"Invoking chain for:\n{query_str}")
result = chain.invoke(query_str)
click.echo(f"\n\nAnswer:\n{result}")




### Trying it out

Now you can run `echo "<your question>" | poetry run python -m dewy_qa query`.

## Conclusion

By following this guide, you've learned how to create a CLI that uses Dewy to manage knowledge and LangChain to process questions and generate answers. This tool demonstrates the practical application of combining a structured knowledge base with the analytical power of LLMs, enabling developers to build more intelligent and responsive applications.

## Further Reading and Resources

- Dewy GitHub repository: [https://github.com/Dewy](https://github.com/DewyKB/dewy)
- Dewy LangChain integration repository: [https://github.com/DewyKB/dewy_langchain](https://github.com/DewyKB/dewy_langchain)
- LangChain documentation: [https://python.langchain.com](https://python.langchain.com)
- OpenAI documentation: [https://platform.opnai.com](https://platform.openai.com/docs/introduction)

Extraction Matters Most

Ben Chambers — Wed, 28 Feb 2024 21:33:32 +0000

There is a lot of content on getting started with Retrieval Augmented Generation (RAG), and numerous posts on various forms of “advanced” RAG. LangChain lists 8 different PDF loaders, 4 different text splitters, and a variety of techniques you can add such as Parent-Document Retrieval and Multi-Vector indexing. It is easy to be paralyzed with doubt – which of these should you use?

If you’re wondering which of these techniques are best for your domain, the long answer is – you should understand which techniques may help your documents and run experiments on your data to see which of those help.

If you're in a hurry, the short answer is that how you extract information matters most. Embeddings and advanced splitting and indexing strategies don't help (or actively hurt) retrieval.

To understand the importance of different factors and prioritize implementation of corresponding techniques within Dewy (the open-source knowledge base we’re developing), we ran some experiments on different configurations. We’re posting these to hopefully save you some time by using these results as a starting point, and possibly using some of the methodology from this post for your own measurements.

Experimentation

There are a variety of frameworks for evaluating RAG and LLM usage, such as TruLens and RAGAS – at the time of writing LangSmith supports evaluation but doesn’t consider the context, making it less suited for evaluating RAG. The general pattern is to record the question and answer (and often the retrieved contexts) and then use an LLM to grade the responses. Typically, you may ask for grades along different metrics – such as “are the retrieved contexts relevant to the question” or “is the generated answer grounded in the retrieved context”. This approach lets you measure the performance across a large set of generated responses, and fits well with the existing tracing systems like LangSmith. For this article, I used RAGAS.

The data set I’m using is the History of Alexnet. This data set includes a PDF representing the source content (the document or documents to index and retrieve) and a set of 160 questions and answers. For your experiments, it’s best to have your LLM application record actual questions and possible golden answers – generated by a human or using a larger model and/or larger context.

All of the frameworks mentioned support running the experiments out-of-band. This allows you to record the actual questions and answers your users are asking, and then run evaluation after the fact to assess the quality of the answers you’ve served up. This is very useful for monitoring your LLM application.

RAGAS provides a variety of metrics. For these experiments, I’m only measuring the following metrics related to retrieval.

Context Precision: Whether the context items relevant to the ground-truth answer are ranked higher than those irrelevant.
Context Recall: The proportion of sentences in the ground-truth answer attributed to the retrieved context. This uses the ground-truth answer as a proxy for the "correct" context.

To simplify comparisons in this post, we use the geometric mean of precision and recall. This is often called the “f1 score”, but since the RAGAS metrics context recall and precision are slightly different, we’ll call this the “context f1”.

See the code for the experiments.

Results

For our experiments, we ran using 3 different extractors (pypdf, pymupdf and unstructured), 5 different chunking and indexing strategies (recursive chunking, parent-document chunking, semantic chunking, and questions-answered multi-vector embedding with both recursive and semantic chunking), 2 different embeddings (OpenAI ada-002 and bge-en-small) and 2 different search strategies (similarity and maximum-marginal relevance). We ran the experiment and evaluation on the cross-product of all of these factors.

The results are available in a zip file and the analysis notebook used to produce the charts is available in Google Colab. For each of the choices we show a heatmap comparing the median of the relative differences Context F1 between the choices.

Extraction

Our suspicion was that extraction would play a significant role in the results. After all, if the wrong text is extracted from a PDF, or it is extracted in a confusing manner there isn’t much that can be done to retrieve the right content.

The results largely supported this. In this visualization you can see that switching from pymupdf to unstructured produces an 87% increase in f1 score, and changing from pypdf to unstructured produces a 72% increase.

Chunking & Indexing

On the indexing front, the results were somewhat surprising.
We expected more sophisticated chunking strategies like semantic to outperform the basic recursive chunking. But the results suggest that recursive generally outperforms the other techniques. As you can see, recursive indexing produces context f1 scores 81% better than semantic, and 56% better than parent indexing.

Generating the questions-answered is a relatively expensive process (applying an LLM during indexing) and didn’t seem to significantly improve upon the baseline chunking.

In the future, we’d like to rerun these experiments on larger datasets to see if these advanced indexing techniques become more important in different cases. Until then, this suggests that for simple RAG use cases, simple indexing strategies like recursive work well.

Search

Many vector store retrievers support basic similarity search as well as a “maximum marginal relevance” or MMR. This is a technique to retrieve more diverse chunks with the hopes of producing richer contexts. We expected MMR to outperform similarity, but in our experiments we saw the opposite - MMR reduced f1 scores by 13%.

Our hypothesis is that because the dataset has a single PDF, there isn’t much benefit to increased diversity. This is another experiment we’d like to re-run on different datasets in the future.

Embeddings

On embeddings, we expected the larger (and better) embeddings from OpenAI ada-002 with 1536 dimensions to outperform bge-small-en-v1.5 which only has 512 dimensions. While there is technically a difference, it is basically insignificant.

Conclusion

While it is important to consider (and measure!) the benefits of these techniques on your own documents and questions, the above analysis suggests that the most important thing you can do for your RAG application is ensure you’re extracting the right content. After that, relatively simple chunking, indexing, and retrieval provides good results.

In the future, we’d like to re-run these experiments with a larger dataset and by feeding the entire set of documents through the models rather than comparing advanced RAG techniques to baseline answers using naive RAG.

Don’t be paralyzed by all the options – nearly any choice you use will allow answering questions over the domain specific data and significantly improve the experience of using your GenAI application. Providing the right context reduces hallucinations and lets you use GenAI to answer questions about domain specific information.

Dewy makes this easy. It provides a RAG service with built-in indexing and retrieval built on the lessons from these experiments and more, allowing you to add documents and retrieve relevant contexts with a simple API and then get back to improving your GenAI application!